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NEAIR Friends and Colleagues: 

The 40'^^ annual conference of the North East Association for Institutional Research was celebrated 
November 9-12, 2013 at the Newport Marriott in the nautical setting of Newport, Rhode Island. While 
these Proceedings include the majority of intellectual content shared at the meeting, they cannot capture 
the spirit of camaraderie, support, professional commitment, and unity that permeated the conference. 

Our conference attendance of 438 ranked the highest ever. It is clear to me that NEAIR is a valuable 
organization of its members. This conference provided professional development, networking and a forum 
to discuss the aspects of institutional research. It enabled our members to strengthen their past and 
current knowledge as we proceed into the future. 

NEAlR's long-standing policy of entrusting the conference planning and execution to a program chair and 
local arrangements chair succeeded in offering a superb conference for all attendees. As program chair, 
Alan Sturtz reached for the gold and compiled a stimulating and rounded set of plenary speakers and other 
presenters. Deborah Santiago instilled us with great enthusiasm in her opening plenary discussion on the 
changing demographic landscape of higher education. Dr. Patrick Terenzini engaged us in exploring change 
and stability in institutional research with lessons learned from a life-time of experience and a look ahead 
at what the future may hold in the profession. The panel of knowledgeable NEAIR members shared 
personal experiences in work-life balance. In addition to our three plenary sessions, our program consisted 
of 17 preconference workshops, 9 contributed papers, 39 workshares, 11 techshares, table topics, special 
interest groups, and 9 vendor showcases. 

As Local Arrangements Chair, Carl Ostermann collaboratively coordinated a committee dedicated to dazzle 
us with the various restaurants and entertainment within the Newport area and an opportunity to 
experience WaterFire in Providence. We feasted on lobster, danced, enjoyed numerous karaoke singers and 
listened to some well-known tunes with the lyrics adjusted to represent our IR experiences. We met new 
friends and colleagues throughout the conference and reenergized ourselves with a walk or run on the 
spectacular Cliffwalk. I think I can rightfully boast and say that the Newport conference was one of the best 
on record. 

I am ultimately thankful for the members of the Steering Committee and Beth Simpson for keeping me on 
track during this past year. I am grateful of each member's suggestions and comments on the policies we 
discussed throughout the year. They provided wisdom and professionalism allowing us to engage in frank 
and open discussions that will ultimately benefit the association's future. 

Finally, my gratitude goes to Tiffany Parker, our Publications Chair, who spent many hours preparing this 
document. Fler work in collecting, editing, and producing the Proceedings in electronic form, is greatly 
appreciated. Fler work has helped us preserve a piece of intellectual history that will be read by many in the 
years to come. 


Catherine Alvord 
NEAIR President 2012-13 
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Aspiring to the role of Data Badass: Some thoughts on the political context of IR 


The genesis of this paper was a grant proposal - sadly, a failed one - to the Spencer 
Foundation a few years ago in a program they use to have called “Data Use and Educational 
Effectiveness” or some such thing. The grant program gave small grants to educational 
researchers seeking to better understand the organizational factors that determine whether good 
educational research and data are actually used in decision-making contexts. My basic research 
design was to conduct a survey of all of you - IR professionals - about specific instances where 
data was used, and where it was not. 

Because perhaps no one in higher education is better positioned than the IR professional 
to understand this basic fact about human nature: people believe things that are not true. Who 
knew? Specifically, people believe things that tend to serve their self-interest - if reality and self- 
interest collide, empirical reality tends to be on the losing end. This is pretty much the core truth 
of my entire training in the discipline of psychology. The study I proposed was, in essence, an 
effort to understand the contextual and organizational factors that reinforce this unfortunate facet 
of human nature, and make our own work as institutional researchers / truth-tellers that much 
more difficult. 

Rejected grant proposals aside. I’m not sure anymore that this is the most fruitful line of 
attack. I doubt that it will ever be possible to define the “ideally” receptive organizational context 
to the kind of work we do. Even if we could, do we have the line authority to create and maintain 
such a context or process? Put another way, do we really want to play the role of umpire or 
referee if we’re not really sure everyone is ultimately going to play by the rules? And, is this 
really the role our senior leadership wants us to play? Umpire? Referee? I used to think that, but 
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when, after data presentations in which I have carefully maintained my objectivity, I am 
increasingly asked for my “point of view” or “what I think”, it feels like I’m being asked to drop 
the referee’s uniform, get off the sidelines, and get on the field of play. 

This is a new role, and it isn’t the one I got into the field of IR intending to play. Nor, I 
suspect, did you. If we are not disinterested, objective “researchers”, but instead find ourselves 
being asked to close the laptop, turn off the Power Point, and pull up a seat at the table to engage 
politically with a “point of view”, what stance should we adopt in this unfamiliar role? This 
paper is a preliminary attempt to flesh out some answers to that question. 

First, for those of you who don’t know me personally, a little personal history is in order. 
I began my career in IR as a painfully naive ex-professor of social psychology, who genuinely 
believed that the sole impediments to having data directly inform decisions were methodological 
or analytic in nature. That’s not just rhetoric - I genuinely did believe this - that the institution 
that hired me was like parched earth on which the flow of data and analysis from my newly 
created office would be a source of sweet water. That the only limitation to the impact I would 
have would be my own skill at divining, and then communicating, “what the data say.” 

It has taken me the last 1 IV 2 years to learn that this is not, in fact, true. For institutional 
researchers, this is not cynicism, but realism. I have had to learn and re-learn this basic reality of 
institutional research the hard way, sometimes embarrassingly, usually in public, with my patient 
employers and indeed with many of you, as my indulgent tutors. 

But I do take these issues of defining our profession very seriously. To hear my wife tell 
it, perhaps a bit too seriously - though many of the themes of the recent IR literature reassure me 
that I am far from alone in perseverating about just what it is we should be doing in this day and 
age as institutional researchers. For verily, I say unto you that the path of the righteous 
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institutional researcher is beset on all sides by the tyranny of rapid changes in technology, 
globalization, data proliferation, not to mention the ongoing restructuring of higher education 
itself. 

For all the talk of “data-driven” decision-making on our campuses and of institutional 
research being a perennial growth area within them, with expanding areas of responsibility and 
resources to match, my own unfailingly pessimistic sensibility directs me to consider another 
possibility. Is it not just as likely that this burgeoning demand for the kinds of services that are 
currently aggregated in our offices may lead to the centralized, “full-service” IR office ceasing to 
exist in recognizable form by, say, the year 2030? 

Well, now that’s a downer. But hear me out, I ultimately have a positive message here. 
Some of you will recall the example from Clayton Christenson’s oft-cited paper Disrupting 
College of how the multi-service firm IBM, just when demand for personal and business 
computing was fully maturing, when it had reached the height of its market dominance, was 
picked apart with astonishing speed by competitors specializing in single areas of its full-service 
model. These more narrowly specialized competitors were able to offer products and services at 
higher quality and lower cost. I think we should be haunted by this analogy, on behalf of our 
“full-service” institutions as Christensen intended but also, closer to home, because the analogy 
fits all too well when applied to the centralized IR professional and the role of his or her office. 
We are the prototypical “Jack (or Jane) of all trades, master of none.” Or, as I learned from my 
daughter’s “365 new words a year” calendar entry, the prototypical “factotum.” A factotum, you 
will be interested to learn if you don’t already know, is defined as “an employee with a diverse 
range of capabilities and responsibilities. A general servant.” 
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At any rate, might the burgeoning demand for “data-driven decision-making”, rather than 
expanding and raising the profile of our roles in their current, familiar form, instead mean fewer 
factotums and more specialists, as other functional areas within our organizations professionalize 
or as outside consultants begin performing parts of the IR function more efficiently? If I’m right 
about the IR role going the way of IBM - that functions performed by the IR office will continue 
to grow in importance, but that they are unlikely to be performed by a single, centralized IR 
office - then several important things follow. 

Most broadly, it means that efforts to define the proper role of our profession - such as 
this one - are not mere narcissistic, self-referential navel-gazing, but are. . .well. . .kinda 
important. How institutional research offices answer questions such as: What is the core role that 
cannot be “outsourced” to other functional areas? Which functions can be “let go” without 
compromising that core? How do we collaborate without making ourselves redundant? will 
determine what we spend our time doing in the coming years, or indeed whether we are there to 
do it at all in our current role. Practically speaking, I think it may mean that we will need to 
specialize - to get better at one important thing, even if it means getting less good at a wide 
range of things. Less factotum, more specialization. 

The good news, I think, is that for the most part our institutions recognize the ungainly 
nature of our roles and responsibilities (a recognition manifested in the run-on titles they’ve 
given our office, among other things, current company included) - and as a result WANT us to 
do this - that is, to help them to think critically about how we can best serve the broader goal of 
helping our institutions become more data-driven. Because, let’s face it, whatever the nature of 
our responsibilities as IR offices today , the creation myth for most of our offices does not feature 
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a wise senior leadership thoughtfully debating how best to align their decision-making and 
resource allocation more closely, using data. 

In fact, the prime mover in the actual creation myth for most of our offices, though 
shrouded in mist, bears the initials are I.P.E.D.S. and U.S. News. That is: compliance reporting. 
Sure, our functions have matured since then, but rare is the institution that can truly say it has 
thoughtfully considered how best to position its IR office so as to become a maximally data- 
driven organization. But I think the time is ripe - our senior leadership wants to have this 
conversation, and wants us to participate in, if not lead it. 

While there is always an element of naked self-interest in this kind of active role 
negotiation, I reject any suggestion that it is in any way opposed to the best interests of our 
organizations. A silver lining, here: perhaps we are lucky to be living in a time and working in a 
profession when our own naked professional self-interest and the interests of our institutions are 
truly aligned to an unusual extent. This will not always be the case, but in the medium-term this 
period when the meaning of “data-driven decision-making” and of the role of IR with respect to 
it are still very much in flux , I think we are most helpful to our institutions when we actively 
advocate for the role that we think best serves that purpose. 

OK - so 12 minutes in, I finally arrive at the thesis statement of my little talk. It concerns 
the role that I think best describes the function in which our “multi-service” offices should begin 
to specialize. Now, I went back and forth on this several times, and can’t help but feel I’m going 
to regret it, but I simply cannot think of a better alternative term for the function or role I have in 
mind for us: that of data bad-ass. 


[slide] 
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Those of you working for institutions with a religious mission may prefer the term data 
champion or data advocate, or even change agent, all of which adequately capture the gist of 
what I’m trying to express here. But not quite: my own carefully considered first preference 
remains data bad-ass. 

Just so you appreciate how hard I tried to come up with an alternative to this very un- 
professional term, I want you to know I actually googled the phrase “bad-ass synonym” 
repeatedly, over several days, while I was preparing this talk. But I kept finding thesaurus entries 
a poor substitute. What finally nailed it for me, was this definition from the unauthorized online 
Urban Dictionary (modified somewhat for family audiences): 

Bad-ass: One with very large ego, which allows them to be direct and ejficient in all they 
do. An indijference to the standards of others, but with a personal moral code... one who 
disregards power in numbers and will confront multiple opponents who defile this 
personal moral code. They are often quiet, but can be outspoken. When they do speak, 
what they say is direct, wise, and simple. 

Unfortunately the popular definition of “bad-ass” is a bit more of a troublemaker than I have in 
mind. More Henry Fonda from 12 Angry Men and less Thelma and Louise. 

[slide] 

So I’ll be more explicit that the recipe for the self-concept that I have in mind is about one-third 
nonconformist / troublemaker, about one-third directness and a “willingness to share unpleasant 
facts when necessary”, and about one-third “formidable skill.” And I’ll grant that for all of these 
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a “very large ego” or at least a very thick skin is a prerequisite. This is the role that I think we as 
IR professionals should cultivate and enact. 

At this point the obvious and uncomfortable question presents itself - does this number 
cruncher at the microphone, with his “very large ego” and apparent delusions of grandeur 
actually conceive of himself this way? As a kind of Samuel L. Jackson with a pocket protector? 
Let me relieve your concern by saying that by no means do I consider myself a fully mature 
specimen - hence the “aspiring” in the title of this talk. But when I look back on the moments in 
my career in which I think have had the greatest positive impact on my institutions, they 
occurred when I made at least gestures toward the high standards of this role: when I had to 
argue for an unpopular point of view; was able to effectively communicate unpleasant, as 
opposed to pleasant, facts; and was most able to demonstrate notable, if perhaps not 
“formidable”, analytic as well as interpersonal skill. I want to take the rest of my time up here to 
share some examples that flesh out what I am talking about. 

The first example is where my latent inner bad-ass was roused into action first by an 
opportunity to exercise a compulsive urge that I am sure is a fa mi liar one for many of you - to 
debunk “accepted wisdom”. Now, nobody likes lazy thinking and untested, sweeping assertions, 
but I use term compulsive here to denote something more - a kind of psychological ‘fit’ , sudden 
in onset and which does not subside until one has rushed back to one’s office, spent the next 
three hours compiling data and doing nothing else until an airtight, compelling analysis 
debunking said accepted wisdom has been prepared and written up. 

Now, I know what you are probably thinking: “Wow, that’s really great, Mark, so you 
actually believe one of our most important functions is not to increase institutional knowledge, 
but to actually undermine, debunk, or otherwise reduce it? Really?” To that unasked question, I 
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answer a cheerful and unabashed, “yes.” When debunking is done with a sense of humility and a 
keen awareness of the fact in the end we are “just the data analysts”, not the ones bearing direct 
responsibility for important bottom-line results, I think this is one of the most important ways we 
can add value, undermine groupthink, and at least delay misguided and costly decisions. 

Enacting this role of “non-conformist debunker” - again, with respect and humility - also serves 
to highlight to the senior leadership as a whole the importance of our office’s independence and 
objectivity from any one functional area. A de-centralized function, in contrast, with “mini-IR 
shops” in each administrative area, is less likely to surface “debunking” data or any indeed any 
information that conflicts with the status quo, whether in theASir own area, where it may be 
perceived as defending (or undermining) one’s boss, or in someone else’s, where it may be 
perceived as partisan or illegitimate (i.e., who asked you, anyway?). 

At any rate, I think this “compulsion to debunk” - and the first bad-ass trait of non- 
conformity and independence - is illustrated by my first data example. Going into the a 2009 
Alumni survey, accepted wisdom in our Development Office was that the class years of the 70s 
and early 80s represented a “lost decade” as far as fundraising. Fewer and less frequent gifts and 
lower reunion attendance were indeed the norm for these cohorts. Our Chief Development 
Officer, though only just having arrived at our institution, “knew” this was a history effect - 
specifically the result of shifting cultural norms about identification with “institutions” generally 
during the tumultuous historical period when these alumnae/I attended our institution. Sure 
enough, our alumnae survey data show a sharp drop-off in “willingness to recommend” - and in 
alumni giving to - our institution for these class years. 

But somewhere around the fifth or sixth time I heard it said that “we know” that the 


dropoff was due to broader forces at work in our larger culture, and had nothing to do with our 
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institution per se at that time, my thoughts became disoriented, my face grew hot, and I knew 
that my compulsion to debunk had kicked into high gear. Rushing back to my office, I was able 
to produce this chart, which I shared, offline, with our CDO. 

Since that time, I have not heard that 
piece of accepted wisdom put forward as 
anything but one of several “possible” 
explanations, one of many others than were now 
focused on the specific history of our institution 
during that time. This new understanding was 
critical as we afterwards entered a fundraising 
campaign with messaging targeted differently by graduating cohorts. 

There were several theories about “why” this pattern was the case, but that’s not 
important here. Instead, let’s examine how this overall reorientation happened. Doing the 
“debunking” offline, with humility and in what I think of as the “spirit of co-discovery,” was a 
key element - I've learned after numerous missteps that the “spirit of co-discovery” is easier to 
embody in a one-on-one, offline conversation than in a public criticism or reaction. But that 
compulsion to get the interpretation right - or at least to avoid getting it wrong - is one that I 
think only an independent, centralized IR person would have enough of a stake in to want to go 
to the bother of gathering, compiling, and - unsolicited - writing up the relevant data. 

My second example - don’t worry, I only have three - also highlights how enacting the 
trait of independence and “indifference to the standards of others” as a kind of disinterested 
third-party is something that I think only a centralized, IR office can convincingly do. The setting 


80% 



Class Year 


here is the college’s discount rate. 
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Our institution’s own discount rate had increased sharply during the economic downturn 
as our yield modeling - performed by an outside consultant, not yours truly, thank God - did not 
properly adjust for the impact of the economic downturn. This understandably greatly alarmed 
our Chief Finance Officer - by definition, discount rates increases cannot go on forever. The 
downturn led to a fair amount of tussling between our CFO and Chief Enrollment Officer over 
what discount rates should be assumed for the five-year budget model. 

The CFO wanted to hold the line, and thus to assume a gradual recovery to “historical 
norms” and then no increases in the out years. Meanwhile, our Chief Enrollment Officer knew 
that was probably unachievable during this national period of increasing tuition and flat incomes, 
and wanted a cushion - or at least a tacit understanding - in the form of an assumption of 
increasing discount rates going forward. The latter was the one who approached me to see 
whether there was not a way of empirically demonstrating her gut sense that holding the discount 
rate flat over the next five year period would be impossible. 

After a lot of thrashing around 
in the dark with national data on family 
incomes, inflation, unemployment, 
projections for high school graduates, 
trying to develop something that could 
reliably project discount rate or net 
price based on these national-level 
variables, I ended up throwing up my 
hands and put forward a very simple historical trend line display. Not formidable skill, by any 
means, but probably the best assumption. 
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But, because the result pleased neither of them completely, it is something that no one but 
an independent, objective analyst would put forward. Again, this rate of increase pleased neither 
the CFO nor our Chief Enrollment Officer, but produced a kind of fruitful detente. An annual 
increase of less than one percent was better than nothing, but it was not something that was going 
to give our Chief Enrollment Officer any kind of meaningful cushion as she tried to craft next 
year’s entering class. Meanwhile, from the CFO’s budget modeling perspective, the increases did 
not impact the budget adversely in the short term - providing some relief in the short term as 
compared to a more inflationary assumption for discount rate - but any assumption of increasing 
discount rate eventually wreaked havoc with the budget model in the out years. Yet the number 
itself was unassailable. For either person to argue with it would mean arguing not only with 25 
years of our own history, but with 25 years of history for our entire sector of the higher 
education industry. 

The important result of this effort was to better align the budget model assumptions with 
reality, to take the guesswork and politics out of this number. A more significant positive result 
was to broaden the terms of this narrow, essentially interpersonal debate to a larger and more 
inclusive discussion about sustainability. This was not rocket science -the data on net price are 
readily available. But having an independent, objective - and assertive - voice in the 
conversation was, I think, essential to this result. Yes, a voice that knows how to find the 
relevant data and analyze in the right way, but more importantly one who simultaneously has no 
direct stake in the data yet understands thoroughly the strategic implications of this number. 
Without that “high-stakes” independence, a central element of this data bad-ass role of IR for 
which I’m advocating, this result would not have been possible. 
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The last example I’d like to share has elements of all three traits, but is mainly an 
example of “being willing to share unpleasant information when necessary.” I’m closing with 
this example because this is the trait of a data bad-ass about which I have the most reservations. 
Yes, the trait of independence and objectivity cuts both ways, but it seems core to our identity, 
even when it means disagreeing; and the trait of having “formidable skill” is something none of 
us could do without. But what about this trait of “willingness to share unpleasant information”? 
Doesn’t this fly in the face of the human tendency to “kill the messenger” of bad news? 
Shouldn’t we be seeking opportunities to emphasize institutional strengths, not harping on 
institutional weaknesses? Thinking solely from the perspective of our own professional self- 
interest, shouldn’t we at least try to maintain a ratio of 50% of the time being cheerleaders 
instead of naysayers? 

One of the many disconnected fragments of wisdom I received from my father in my 
childhood relates to this question. My father was a mechanical engineer - he’d be so proud to be 
cited here; then again, maybe not - at any rate his advice was something to the effect of: 
“Organizations are full of critics, naysayers, and other malcontents who spend their time saying 
‘no’ and trying to explain why something can ’t be done. The people who get promoted within 
the organization are the ones who know how to say ‘yes’ and actually go about delivering 
results.” Sound advice - foolish to try to take that one head-on. So in working around it I’ll have 
try to ambush it from the side. 

Institutional researchers are not like engineers or other kinds of employees. Engineers are 
fault-finding machines - don’t even get me started about how this affected my childhood - but 
ultimately they are paid to build things that work. All of that scrupulous attention to what’s 
wrong with an evolving design is in the interest of this goal. Now, the fault-finding detail- 
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orientation is shared by of institutional researchers, but it isn’t at all clear that our job 
responsibilities always extend to the next step of building things that do work. In some sense, 
then, maybe this wisdom about relative merits of a “can-do” versus a “can’t do” orientation to 
one’s effectiveness and professional advancement doesn’t always apply to IR. 

Now I’m not arguing that the trait of “being willing to share unpleasant information” 
implies that this is all one should do, but rather that it is something that our role, more than any 
other within our organizations, is optimally positioned to do. Trying to maintain even a 50/50 
ratio of “can-do” to “can’t-do” information might find us - as well our institutional leaders - 
ignoring a lot of mission-critical blinking yellow and red lights on our institutional dashboards. 
And while I don’t expect my President, my Provost, or any other senior leaders to regularly come 
knocking on my door eagerly seeking my latest harvest of “unpleasant information”, I do know 
that they do not keep me on the payroll for the purpose of sugarcoating or otherwise concealing 
unpleasant realities. 

That said, this does present our profession with a real dilemma. Success has many 
fathers, while failure is an orphan - this is human nature - but since the role of adoptive parent 
often falls to IR, a focus on what is best for our institution puts us more often in the role of - at 
best - constructive critic than it does of team-playing cheerleader. This choice between honoring 
our obligation to our institutions by sharing information about things that are not working, or are 
not going well - and thus associating one’s office with negativity or worse, with not being a 
“team player”, is a difficult and dismayingly frequent one in my world, and I expect in yours as 
well. But since this is an unavoidable aspect of what we are indeed expected to do, we might as 
well embrace it. In fact we might as well seek opportunities to be explicit about it when we are 
playing this role, and about why it is necessary and valuable that we do so. 
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So this brings me to my final data example, an unpleasant fact about the overall extent to 
which alumnae/i at a former institution I worked at said they would recommend our institution to 
others. At this institution, the single greatest driver of “recommending” among all altitudinal 
items was satisfaction with social life. And that basic finding held true when analyzed from any 
number of analytic perspectives. But that’s not the important thing here. 

In fact the senior leadership was well aware that students never chose their institution for 
its fun-loving party atmosphere, active dating scene, or cuddly and caring faculty and senior 
administration. But it had operated more or less on the principle that that was OK - 1 think some 
may have even taken a secret pride in it and linked it to fundamental part of our culture, as a sort 
of minor side effect of our academic rigor. In any event no one seemed to think it mattered all 
that much to our reputation as a desirable place to learn and spend one’s college years. The 
Alumnae/i survey analysis directly challenged this assumption and, for some, indirectly 
challenged the core of our institution’s identity, because of two basic findings: 

(1) The primary driver of alumni giving was “recommending” 

(2) The primary driver of recommending was one’s level of satisfaction with social life. 
At least at this institution, this latter relationship was so robust that, in effect, asking a graduating 
senior or an alumna whether s/he would recommend their institution was, for all intents and 
purposes, virtually the same thing as asking them whether they had a satisfying social life while 
a student there. 

There were happy stories in our alumnae/i data, of course, and in point of fact even this 
result was an “unpleasant fact” only as a matter of direction and not in degree - most alums were 
satisfied and giving rates were fine - but my willingness to highlight that essentially negative 
finding as the most important and “actionable” one turned out, to my surprise, to be critical. For, 
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as it turned out, despite the cultural veneer of relative indifference to students’ social well-being 
- more colloquially, for their opportunities to have “fun”, there was a revolution waiting to 
happen. 

A coalition, if you will, of trustees, administrators, and faculty seemed to have been 
waiting for some kind of excuse or permission to facilitate student’s efforts to lighten up a bit 
and have “fun” as well as scholarly rigor. This reaction was not immediate, nor was it uniform, 
but I think my willingness to continue to be direct about this “unpleasant” aspect of our 
alumnae/i data played at least a small role in several initiatives focused on improving student’s 
social experience since that time. This was perhaps best exemplified by the inclusion of efforts to 
enhance the student experience as one of the four major, highest-level goals in this institution’s 
strategic plan. I think I can say that without this data, that part of the strategic plan would be 
absent. 

So what have I been up to here? Notably, this talk and my data examples have 
downplayed the technical role of IR in producing quality data and quality analysis, arguing 
instead that the important factors that determine whether data and analysis “have an impact” are 
for the most part interpersonal and contextual. And that, in fact, the major variable over which 
we have control - the role and communication style we adopt when we present the data - is 
perhaps the most important. That is not to say I don’t think the actual data and presentation of it 
were irrelevant to that impact. Just having finished a do-it-yourself kitchen renovation, I am 
experiencing first-hand how much the capacity to “do it yourself’ with any degree of quality 
depends on having adequate - if not formidable - skill, and on good tools. So, yes, data and 
analysis matter. But maybe they are not enough anymore. 
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I would argue that the most under-appreeiated and underutilized resource of our 
profession is the collective wisdom of data bad-asses here in this room regarding the not-always- 
comfortable-to-talk-about interpersonal and contextual factors that finally determine how our 
data tools are actually used on our diverse campuses. That is the spirit in which I offer this talk 
and its proposal of the “data badass” role as my own small contribution to this collective 
wisdom. I hope you found something useful in it, in the data examples, but equally in the context 
surrounding them, and in what they suggest about the core attributes of an IR function that will 
remain valuable - and employed - in the years to come. 

By way of ending, I want to leave you with one of my favorite jokes - which came to me 
again courtesy of my retired mechanical engineer father - that captures why I think we need to 
exert more of our collective energy toward the task of better defining the unique and non- 
outsource-able nature of our roles. 

Here it is. So a homeowner has a clogged sink drain, and calls in a plumber. Clearly I still 
have kitchen renovation on the brain. Anyway, the plumber walks in, looks under the sink for a 
minute, heads down to the basement, feels around the drainpipe a bit, and then gives the 
drainpipe a gentle tap with his hammer. There’s a small gurgle and then a sudden “whoosh” as 
the drain clears. The plumber turns to the homeowner and hands him an invoice on which is 
scrawled. 

Repaired clogged drain: $100. 

After picking his lower jaw back up off the floor, the homeowner says, “What?!? You’ve 
been here less than one minute and all you did was tap that drainpipe with your hammer! That’s 
not worth charging me $100!” and hands the invoice back to the plumber in disgust. Plumber 
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pauses, scratches head. Crumples invoice and stuffs in pocket, writes a new one, hands it back to 
the homeowner. It reads: 

Tapped hammer on drainpipe to repair clogged drain: 25 cents 
Knowing where to tap: $99.75 
Total: $100 

So do our institutions view our offices as hammer tappers or as invaluable resources of 
practical knowledge, uniquely positioned to be able to identify and communicate the most 
actionable bits of insight to those who may not always want to listen, or who may have an active 
stake in not listening? The answer will of course depend on having access to good analytic tools 
— on rich sources of information and on our skillful analyses of them. But it will depend just as 
much, if not more so, on how well we can tap into our inner data bad-ass, and effectively 
navigate the political context in which that data is used. 
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First NEAIR Paper 


Title: Data-Driven Internal Benchmarks and Successful Learning Outcomes 
Authors: Mamta Saxena and Douglas L. Florfrom Northeastern University 

Objectives: Use institutional data to augment student learning outcomes as part of the program assessment and refinement initiative. 

Intended Outcomes for Attendees: 

1. Assess the value of using Key Performance Indicators (KPIs) based on student success data (as defined by NEASC, S-Series data) for 
assessment and refinement purposes at the program level 

2. Establish internal benchmarks for KPIs of student success 

3. Derive and establish statistically valid signal values to inform programs of both quantitative and qualitative evaluation of KPIs for 
refinement of instructional practice at the program level 

4. Report individual institutional programs comparisons with the internal benchmarks and specific signal values to: (a) track progressive 
refinement and, (b)augment student outcomes 

Literature Review: 

Six years ago, the Spelling Commission report (2006), "A Test of Leadership: Charting the Future of U.S. Fligher Education," called attention to 
the fact that the quality of student learning in U.S. is inadequate and, in some cases deteriorating. One of the major criticisms was that higher 
education degrees are not well-aligned with the competencies that are expected of college graduates. Since then, there have been numerous 
debates on related issues in higher education such as persistence and student retention, poor alignment of degree programs with workforce 
requirements in the professional field, and the inadequate assessment of student competencies. 

In a supplement document released after the State of the Union address on February 13, 2013, the Obama administration proposed the biggest 
change to federal higher education policy since the Fligher Education Amendments of 1972. President Obama called for benchmarks to be set for 
affordability and student outcomes as criteria for receiving federal student financial aid. These benchmarks aim to reshape American higher 
education by modifying the accountability system in regard to cost, values, and quality. According to Ambrose (2010), "to develop mastery, 
students must acquire component skills, practice integrating them, and know when to apply what they have learned." Likewise, employer 
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surveys highlight the increased demand for a quality workforce for the 21rst century knowledge economy: college graduates with not only 
degrees reflecting meaningful learning, but also the ability to apply a variety of acquired interdisciplinary skills such as critical thinking, problem- 
solving, and the ability to learn on the job. 

In the 1992 revision of the Standards for Accreditation and the implementation of the Policy Statement, NEASC defined institutional 
effectiveness as "the capacity of an institution to assess, verify, and enhance the fulfillment of its mission and purposes, giving primary focus to 
the attainment of its educational objectives." Assessment of student learning, according to the Policy Statement, is a key indicator of 
institutional effectiveness. The statement further affirmed that, "while assessment is an overall institutional concern, as reflected in the various 
standards for accreditation, its primary focus is the teaching-learning experience. To the greatest extent possible, therefore, the institution 
should describe explicit achievements expected of its students and adopt reliable procedures for assessing those achievements." 

Besides accreditation, the changing landscape in higher education constitutes many challenges and opportunities for assessment practices 
including governance, financial aid, tenure, standardized testing, MOOCs, competency-base models, and the critical need for workforce 
alignment with college degrees, to name a few. Therefore, colleges and universities that grant degrees have the responsibility to assess and 
improve the quality of student learning and ensure that graduates acquire the required knowledge, skills, values, and attitudes toward the end 
of the degree program that will prepare them for work, life, and responsible citizenship. There is increased internal and external pressure for 
accountability in higher education in regard to student success and learning outcomes. How can institutions answer the call for accountability 
and fulfill the expected responsibilities? 

The aforementioned question has a direct correlation with the framework used by higher education institutions to assess curriculum. More 
importantly, the issue at hand begs the question, how can institutions efficiently use the assessment data to inform refinement practices and 
improve student learning outcomes. Campbell (2007) quotes several statistics to make a case for current conditions in higher education still at 
risk (as stated 30 years ago in the "A Nation at Risk" report released by the U.S. Department of Education in 1983). He defers to academic 
analytics as "a new tool for the new era", to respond to the accountability paradigm in higher education. 

Although there is no magic formula for demonstrating institutional effectiveness or student learning, assessment efforts at College of 
Professional Studies must align with Northeastern University's mission : "(a) to educate students for a life of fulfillment and accomplishment, 
and (b) to create and translate knowledge to meet global and societal needs." Furthermore, assessment should not be looked at as a one-time 
activity to fulfill NEASC standards but an ongoing, integrative, and organic endeavor to help Northeastern University reach its maximum 
potential in achieving its mission. As mentioned earlier, one of the primary focal points outlined in the accreditation standards is the teaching- 
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learning experience. Assessment efforts should, therefore, be embedded as an integrated element of the learning experience and curriculum 
from start to finish and thereafter: a progressive refinement model, not just an "add-on" to the curriculum. 


The "Principles of Good Practice for Assessing Student Learning" released by the American Association of Higher Education's Assessment Forum 
have matured overtime and have been used widely as an effective tool for educational improvement. These first principles emphasize the 
significance of a clear purpose, ongoing efforts for improvement, and the inclusion of multiple stakeholders in the policy and decision-making 
efforts. As we began to think about promoting a culture of assessment within CPS and working with other assessment and quality assurance 
teams at NU Online, we needed to adopt a collaborative approach to implementing the first principles into our assessment practices. The table 
below outlines the implications of these principles for the design practice. 

Table 1: Implications of First Principles for Assessment 


First Principles 

Implications 

Assessment should begin with the institution's vision of the types of 
learning most valued for students to achieve in the course of study 

Define program competencies based on the institution vision, mission, and 
values 

Assessment should reflect the multifaceted and enduring nature of 
learning as demonstrated via performance over-time: knowledge, 
abilities, attitudes, and habits 

Define concrete outcomes and objectives at program, course, and module 
level 

Assessment works well with clearly stated purposes 

Map and sequence courses to program level outcomes and competencies: 
curricular mapping 

Assessment requires attention to both outcomes and relevant 
experiences 

Design appropriate assignments and activities for formative assessment and 
authentic, summative assessment at course level 

Assessment leads to improvement when it involves participation from all 
stakeholders and representatives, asks questions that people care about, 
and embraced as a culture institution wide 

Promote a culture of assessment by establishing a shared framework and 
communicating and aligning efforts across departments 

Assessment is truly valuable when used as a medium to report accurate 

Use the data collected from assessment to enhance design, facilitate faculty 
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information as part of accountability toward students and society for the 

development, and improve student learning 

sake of improvement 



Walvoord (2010) defines assessment as "the systematic collection of information about student learning, using the time, knowledge, expertise, 
and resources available, in order to inform decisions that affect student learning." (p. 2). The natural steps in assessment include: 

1. Goals: What we want students to be able to do when they complete the degree? 

2. Information: How well are students achieving the goals: the evidence of learning? 

3. Action: How can we use the information to improve learning? 

4. Subsequent Analyses: Assess whether the actions taken had the desired effect, thus "closing the loop" 

The table below uses the aforementioned criteria defined by the four steps in the assessment process and the first principles mentioned earlier 
to craft the implications and areas of development for online assessment practices. 

Table 2: Assessment Plan: Steps and Implications 


Steps 

Implications 

Areas of Development 

Goals definition: What 
are we measuring and 
against what? 

If the purpose of assessment is to answer the question, 
"are students learning what we want them to?" we need 
to define the "what" in a concrete manner and ensure that 
the curriculum (outcomes and assessments) is well-aligned 
with the institution's mission and program outcomes 
(competencies). 

• Establish a shared vocabulary for outcomes and 

objectives 

• Define concrete program and course outcomes 

• Define tools for curriculum mapping- two dimensional 

matrix of courses (outcomes and assessment) 
against competencies 

• Establish a framework for course level definition of 

competencies: Rubrics and Assignment templates 

Information collection: 
What is the evidence or 
measure of learning 
and how are we 
documenting/recording 

Before we collect data we need to define the curricular 
sequence in an intentional and cumulative fashion so we 
can define the sequence of competencies to be mastered 
and assignments to do at the program and course level. 
We need to document learning via individual data 

• Complete curriculum mapping at program and course 

level by locating signature assignments/assessment 
items across the program 

• Create course maps, rubrics, and assignment templates 

at course level to answer the student's question: 
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that data? 

collection at program and course level. 

Why do 1 have to take this course and how is the 
assignment aligned with the outcomes and 
competencies in the degree program? 

• Create systems/dashboards to document/record student 
performance against competencies and outcomes: 
Which competency is mastered at what level, what 
point in time, demonstrated by what kind of 
assessment? 

Action: How can we 
use the information for 
progressive refinement 
in design? 

Some of the drawbacks of assessment include focusing on 
compliance with standards, gathering data that is not used 
for decision-making, and making the process too complex. 
When done right, assessment leads to actions such as 
changes in programmatic structures, requirements, 
policies, funding, planning, faculty development, enhanced 
design, and ultimately, improved student learning. Hence, 
we need to document learning not only via individual data 
collection at program and course level, but also by 
defining and creating systems for feeding that data into 
decision-making. We can, however, build on what we 
already have as evidence of learning and keep it simple! 

• Create systems and processes for analyzing data to 

create internal and external benchmarks 

• Create systems and processes for feeding the analysis 

results and benchmarks into design decisions (act 
upon the data) 

• Create systems and processes for tracking the results of 

the actions 


Methodology and Data Sources: 

As a means of evaluating indirect measures of program level student learning outcomes as per NEASC (see Student Success or S-series data 
reporting), student data across the years 2008 to 2013 for the College of Professional Studies students was pulled from Banner relating to 
Graduation and Persistence rates, as well as Course Completion rates. Data were analyzed separately for the two levels of students - 
undergraduate students (n=3379) and graduate students (n=3305). Data were then analyzed via the establishment of internal benchmarks for 
these three Key Performance Indicators (KPIs) at each level of student. In addition, specific signal values were set which allowed each program 
to be compared and evaluated with respect to the students within their respective program level. 

Using the mean and standard deviation over a three year period, signal values establish four separate groups for the KPIs associated with 
Graduation and Persistence rates as internal benchmarks. Those programs with values below the mean-minus-one standard deviation were 
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classified as Red, Well-Below Average. Programs with values from the mean-minus-one standard deviation to the mean were classified as 
Yellow, Below Average. Programs with values at, or greater to the mean and yet less than the mean-plus-one standard deviation were classified 
as Green, Above Average. Those programs with values at or greater than the mean-plus-one standard deviation were classified as Blue, Well- 
Above Average. 

Programs in which Graduation and Persistence Rates were clearly classified as trending to the classification of Blue (Well-Above Average) or 
maintained a classification of Blue for more than one year were identified. Programs that were clearly classified as trending to the classification 
of Red (Well-Below Average) or maintained a classification of Red for more than one year were also identified. 

For Course Completions (Withdrawals, Failures, and Incompletes), five separate sets of signal values were developed. If a student successfully 
completed 100% of their courses for a given year, student data were categorized as Blue. If a student did not successfully complete 1% to 20% 
of their courses for a given year, student data were categorized as Green. If a student did not successfully complete 21% to 50% of their courses 
for a given year, student data were categorized as Yellow. If a student did not successfully complete 51% to 99% of their courses for a given 
year, student data were categorized as Red. If a student did not successfully complete 100% of their courses for a given term, student data were 
categorized as Black. 

Data were analyzed separately for Course Completions at each program level (i.e.. Undergraduate or Graduate) for each of the four KPIs (i.e.. 
Withdrawals, Failures, Incompletes, and total number of W-F-ls). Thus, CPS students categorized as Blue or Green indicated that these students 
successfully completed the vast majority of their courses for a given year. Students categorized as Red or Black signified that the vast majority of 
the courses taken in a given year were not completed successfully. 

Results: 

With respect to rates for Graduation and Persistence, each of the Undergraduate and Graduate programs were provided their own respective 
data. Sample paragraphs were written to provide Academic Program Lead faculty (APL) for each program to use as a model for writing up 
results that were included in their Academic Quality Assurance Annual report. APIs thus were allowed to note specifically, not only the 
quantitative position on the continuum with which they found their respective program, but were also able to qualitatively determine where 
their respective program was in relation to the entire program through the use of an internal benchmark. 

With respect to the four KPIs associated with Course Completions, percentages of students in each of the categories were provided to each of 
the APIs for their respective programs and KPIs were also provided for the program level in which they were located (i.e.. Undergraduate or 
Graduate level). This allowed for each academic program to not only note quantitatively their respective position for these KPIs, but also 
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allowed for comparison a qualitative comparison with the internal benchmarks provided. Samples paragraphs were provided to each of the 
APIs as a means of providing a model for writing up the results for inclusion in their Academic Quality Assurance Annual report. 


Additional support and guidance was provided for how to evaluate and develop a Progressive Refinement Plan (PRP) in which the results of 
these data could be used for improvement of the program. PRPs required that specific faculty be assigned and dates for completion of the 
plans. Each of the program write ups that included these indirect measures of Student Success were then reviewed by the Academic Quality 
Assurance (AQA) staff and feedback was provided to the APIs and revisions were made, as needed. Qnce the AQA review was complete, AQA 
Annual reports were then sent to the Academic Deans for the respective program level for approval. 

Example of Results and Progressive Refinement Plan for the MS in Digital Media 


The Digital Media program's Graduation Rate from FY 2008 through 2011 ranged from a low of 38% in 2011 to a high of 69% in 2009. 
Even with the lower percentage in 2011, the program presented percentages greater than two standard deviations above the baseline 
for all CPS graduate programs, whose mean during the same time period was 27%. When you combine the data for Graduated Student 
rates with those for Active Students during the same period, the Persistence rates can be derived. The Digital Media program did not 
graduate its first students until 2008. In 2009 and 2010 the Persistence rate was uneven, being slightly above graduate program average 
of 72% in 2009 (a rate of 75%), and significantly below it in 2010 (a rate of 55%). However, in the period from 2011 through 2013, the 
Graduate Program in Digital Media consistently exceeded the baseline CPS graduate program mean for the timeframe (from 81% to 86% 
for the program compared with the mean of 72%). 

Therefore, with respect to this Key Performance Indicator, the Digital Media program was one of the top performing graduate programs 
in the College during this period. Qverall, this indicates that student Persistence rates have not only trended an increase, but remained 
consistently high relative to the mean for the period from 2011 to 2013. 

Based on the results of the Indirect measures of Student Learning, the program has excellent persistence, but comparatively has a low 
graduation rate. This could be due to a number of factors. 

students are choosing to stay in the program longer to complete courses from a second concentration 
students may be having problems completing or preparing for their thesis projects 

To determine what these results mean, Cynthia Baron will initiate a project and contact Matt Henderson by end of July to request an 
Excel spreadsheet that would contain all Digital Media students, the courses that they have taken, what concentration each has 
declared, and which students are still in the program vs. those who have graduated. 
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Conclusion: 


For the first year of the newly implemented AQA Program Evaluation plan, much success was noted (and documented). Specifically, all 
Undergraduate and Graduate academic programs were provided S-Series data (see NEASC definitions) for use in tracking and evaluating Student 
Success. Academic programs were also provided with historical data to assess trending of Student Success across multiple years of data. 
Providing individual academic programs with program level data in conjunction with establishment of signal values allowed for quantitative and 
qualitative assessment, evaluation, and comparison with internal benchmarks. 

Ensuring the use of these data and the results of each program for use in development of Progressive Refinement Plans in AQA Annual reporting 
ensures that data decision making strategies are used and implemented at the program level. Thus, the use of KPIs and signal values with 
internal benchmarks afforded APIs with the insight of where their respective programs were with respect to Student Success, as defined by 
NEASC, as well as the opportunity to close the loop with respect to subsequent evaluations of the programs' Progressive Refinement Plans. As 
programs implement PRPs with success, the signal values, in conjunction with internal benchmarks are expected to gradually rise over time, 
thereby highlighting the fact that continuous improvement and refinement is truly an ongoing process and not an endpoint. 

Implications for Future Research or Current Practices: 

1. Holistic approaches to program evaluation that use of statistically derived signal values for KPIs can be used within colleges and 
universities to identify lessons learned from well-functioning programs to inform under-performing programs and identify areas for 
progressive refinement. 

2. Descriptive analytics can further help inform the practices of various sectors within the institution: instructional design, enrollment, 
advising, academic support 
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GOING TEST-OPTIONAL: 

GATHERING EVIDENCE AND MAKING THE DECISION 
AT ITHACA COLLEGE 

Yuko Mulugetta, Ph.D. 


Director of Enrollment Planning 
Division of Enrollment and Communication 
Ithaca College 


Introduction 

In recent years, a growing number of colleges and universities^ have adopted a “test-optional 
admission policy” that allows students to opt out of submitting standardized test scores as a part 
of their admission applications. In 2012, Ithaca College joined the group and implemented a 
test-optional policy for the admission applications of the fall 2013 entering cohort. Ithaca 
College is a mid-sized four- year comprehensive private college in central New York. The 
College promotes experiential and integrative learning by blending performing arts, liberal arts 
and professional programs primarily for undergraduates. 

Eittle research on this subject has been done by small or medium-sized comprehensive 
private institutions like Ithaca College. In fact, this type of school might be best suited for a test- 
optional admission policy. The present study is an effort to provide an institutionally- specific 
research example to such institutions that are considering implementation. 

The study includes a literature review, an explanation as to why Ithaca College’s enrollment 
management team decided to propose a test-optional policy, and the step by step research 
methodology. The study also provides a discussion of the research results that played a pivotal 


role in gaining institutional approval for moving the College to the test-optional policy. Easily, 



the results gathered from the first test-optional cohort that enrolled in fall 2013 and conclusions 
for consideration in the future are presented. 

Literature Review 

The controversy over the validity of the use of standardized test scores in the college 
admission process is nothing new. The early intent of the creation of the SAT was to open the 
doors of higher education to students without traditionally- valued credentials; the objective 
testing scheme was seen as a way to “level the field”. Along with this goal, colleges and 
universities also saw standardized testing as a way to enhance their prestige by showing that their 
students were highly qualified based on the test results — not based on social class or connections 
(Epstein, 2009). The premise that standardized testing can effectively identify qualified students 
and accurately predict their future academic success justified use of these tests and led to them 
dominating the college admissions world in the latter half of the 20th century. 

This premise, however, has become subject to severe scrutiny in recent years. The main 
criticism is that standardized tests are culturally biased against subgroups including racial 
minority groups, females, first generation students, and those from low-income strata (e.g., 
Zwick, 2004, 2007). Empirical studies have revealed that female students’ SAT math scores are 
lower than males by one-third of a standard deviation while Latinos’ and Afro Americans’ scores 
are lower than whites by two-thirds and one standard deviation respectively (Rosner, 2012). 

The critics argue, therefore, that standardized tests structurally maintain — or worse augment — 
the already existing gap between advantaged and disadvantaged applicants, by imposing “a 
devastating impact on the self-esteem and aspirations of young students” (Atkinson, 2001). 



Furthermore, it has been argued that standardized test measures are not only culturally 
biased, but that they also may not be the best predictor of future academic achievements in 
college. The studies have consistently found that SAT scores do not predict the college first- year 
GPA as effectively as other measures such as high school GPA or AP credits (e.g., Cornwell, 
Mustard, and Van Parys, 2012). The College Board research team has examined the incremental 
validity attributed to SAT scores over high school GPA (HSGPA) in predicting the first-year 
college GPA (FYGPA). The study used a large cross-sectional sample of data from the 2006 
cohort who took the revised SAT with the newly added SAT writing section. They found that 
when HSGPA was taken into account, the incremental validity attributed to SAT was 0.08, 
which was lower than the incremental validity associated with HSGPA over SAT scores (r = 
0.09). Because of these results, they recommended that colleges use both HSGPA and SAT 
scores to make the best predictions of student success (Kobrin, Patterson, Shaw, Mattern, and 
Barbuti, 2008). 

An increasing amount of evidence, therefore, suggests that the additive power of 
standardized test scores in predicting students’ performance in college is smaller than was once 
believed when high school GPA or AP credits are taken into account. However, the majority of 
this research evidence is provided by large testing agencies (NACAC, 2008), selective large 
public (e.g., Cornwell, et al, 2012), private research universities (e.g.,Wonnell, Rothstein, and 
Latting, 2012) or selective liberal arts colleges (e.g., Rask and Tiefenthaler, 2012). Little 
research has been conducted by smaller comprehensive colleges like Ithaca College despite the 
fact that such schools might be best suited for adopting a test-optional admission policy. The 
present study is an effort to provide a research example to this type of school in deciding whether 
or not to implement a test-optional policy. 



Ithaca College’s Test-Optional Policy Proposal 

In 2009, the College’s internal study revealed a high eorrelation between Ithaea College’s 
admission application numbers and the number of public high school graduates in Northeast. It 
was forecasted by the National Center for Educational Statistics (NCES) that the Northeast 
market would shrink by more than ten percent between 2008 and 2018 (NCES, 2009a). 

In 2009, the College decided to strategically position itself for breaking away from the 
predicted rapid decline of the high school graduate population in Northeast. The strategies laid 
out include the launch of an integrative marketing campaign around the “Ready” theme, and the 
strategic increase in financial aid to enroll more desirable students while raising tuition at a 
slower pace. Eurthermore, the enrollment management team proposed a test-optional admission 
policy in order to increase applications not only from its primary markets, but also from more 
racially diverse communities. Approximately 15% of Ithaca’s freshmen class was from the 
AEANA (African-American, Eatino/a, Asian and Native American) communities in 2009 while 
the institutional plan aimed to grow the ANAEA student population from 15% to 20% by 2020. 

Research Goals 

The following three research goals are formed. The first goal is to examine how well SAT 
math, SAT critical reading and SAT writing scores could explain students’ academic 
performance in college after controlling for the effect of non-SAT indicators such as high school 
GPA or AP credits. In other words, this study wants to compare the College’s results with those 
of previous studies in terms of the incremental validity associated with standardized test scores in 
predicting students’ college performance after taking other effects into consideration. If this 



study finds an insignificant incremental validity of test scores, this would be supporting evidence 
for instituting a test-optional policy. 

The second goal is to analyze and evaluate a crucial difference between this study and others 
regarding how to measure the effect of non-SAT indicators. The majority of previous studies 
have indicated “high school GPA” or “number of AP credits taken” as non-SAT measures used 
for college admission {e.g., Cornwell et al., 2012). Smaller Colleges like Ithaca, however, often 
utilize a more personalized approach. For example, Ithaca College’s admission office is 
committed to the “holistic” application review process, meaning that reviewers make an 
admission decision by evaluating a student’s all-rounded ability with various measures such as 
high school GPA, class rank, transcripts, the profile of high school attended, recommendations, 
essays, extra-curricular activities, leadership skills, evaluation from recommended on-site 
interviews, and audition scores for music and theatre candidates, in addition to standardized test 
scores. In this study, an additional consideration is introduced. To capture a component of the 
“holistic review”, a numerical variable called “Strength of High School Schedule” is created, 
which is a reviewer’s evaluation on how much a student has challenged him or herself in a broad 
array of learning at high school. Since “Strength of Schedule” scores were not originally 
recorded in the computer system, reviewers were instructed to re-evaluate students’ admission 
materials randomly selected from the fall 2007 entering cohort and to record “Strength of 
Schedule” scores on a ten-point scale in a Microsoft Access database created for the present 
study. Details are discussed in the following methodology section. 

The last goal of the current study is to demonstrate that a valid research study can be done 
even with a smaller sample size. Previous studies used very large data sets with over 3,000 cases 
(e.g., Wonnell, et al., 2012). While it is true that the larger the sample, the smaller the sampling 



error, small-sized colleges may not have a large amount of historical data ready for analyses. 

The present study demonstrates that valid research results can be obtained from approximately 
500 sampled cases (see below for details) as long as an appropriate sampling methodology is 
applied. 

Research Methodology and Data 

To ensure the objectivity of research, the research procedure was established by a cross- 
division project team including the Vice President of Enrollment and Communication; the 
Director of Enrollment Planning, the Director and Associate Director of Admission, and the 
Director and other members of Institutional Research. 

The research methodology was well documented by the original study (Borch and Mulugetta, 
2010). The project team decided that the study should focus on fall 2007 first-time full-time 
freshmen who were retained at Ithaca College to their fifth semester, a total of 1,387 students at 
the time of this study. Eurther, the decision was made to take a random sample of 520 (500 
plus 20 additional in case of some missing data problems), stratified by gender and school when 
they entered the College as first-time freshmen. This sample size was chosen with the 
understanding that it would result in a sampling error of approximately 4% (Suskie, 1992). 

Since the College has not established a data warehouse or an enterprise content management 
system, old students’ records are still kept in paper form. Thus, photocopies of the sampled 
students’ high school transcripts were obtained from the paper records of the Registrar’s Office 
and reviewed by members of the Admission Office staff who are typically involved in the 
applicant review process. The results of this high school transcript review for each student were 
entered by the Admission staff into a Microsoft Access database created by Institutional 



Research. The high school transcript data were then matched using SPSS to profile data already 
available in the fall 2007 opening enrollment files of Institutional Research as well as other data. 
Review of the 520 sampled students’ high school transcripts revealed that 48 students should be 
dropped from the study due to missing or incomplete transcripts. An additional 4 students were 
excluded from the study because these cases show extremely large residuals above 3 standard 
deviations in the preliminary regression analysis. Thus, the total number of students included in 
this study’s final analysis was 468. A complete breakdown of numbers and proportions of 
students by original IC school and gender who were in the initial population and in the final 
study analysis are provided in Table 1. The sample is slightly skewed to males in comparison to 
the population. However, overall the sample used for the final analysis is judged to be a 
reasonable representation of the population. 

A list of variables studied is presented in Table 2. The majority of the variables listed are 
self-explanatory. However, two variables deserve special attention. As mentioned earlier, 
“Strength of High School Schedule” on a 10-point scale, measures a reviewer’s evaluation of 
how much a student has challenged him or herself in a broad array of learning at high school. 

The intent was for this variable to capture a component of the “holistic” admission review 
process to which many small colleges are committed. Unfortunately, such measures have not 
been included in previous studies as pointed out earlier (Sternberg, 2012). In addition to 
conventional non-SAT measures such as high school GPA, AP credits and high school percentile 
rank, the inclusion of “Strength of High School Schedule” might reveal the importance of the 
holistic admission review process to predict a student’s success in college, which might further 
solidify the argument for a test-optional policy. 



The second significant difference is that previous studies most often used first-year GPA as a 
dependent variable whereas this study uses the cumulative IC GPA at the end of the 6^*^ semester 
which is, we believe, a more stable measurement of a student’s long-term academic performance 
in college. Some previous studies found that high school grades are better indicators of grades 
beyond the freshman year in college than admission test scores {e.g., Geiser, 2007). 


Table 1: Students in the Population vs. in the Analysis 


Fall 2007 First-time Full-time Freshmen Retained to 5th Semester 

Females 


Ithaca College 
School 

Population 

Sample in Analysis 

Hdct 

% of Total Population 

Hdct 

% of Total Sample 

Business 

62 

4.5% 

21 

4.5% 

Communications 

172 

12.4% 

60 

12.8% 

HSHP 

187 

13.5% 

62 

13.2% 

H&S* 

337 

24.3% 

112 

23.9% 

Music 

60 

4.3% 

20 

4.3% 

Female Total 

818 

59.0% 

275 

58.8% 


Males 


Ithaca College 
School 

Population 

Sample in Analysis 

Hdct 

% of Total Population 

Hdct 

% of Total Sample 

Business 

104 

7.5% 

34 

7.3% 

Communications 

107 

7.7% 

38 

8.1% 

HSHP 

101 

7.3% 

37 

7.9% 

H&S* 

202 

14.6% 

65 

13.9% 

Music 

55 

4.0% 

19 

4.1% 

Male Total 

569 

41.0% 

193 

41.2% 




Grand Total 

1387 

100% 

468 

100.0% 


Note: H&S includes a small number of students in the Division of Interdisciplinary and International Studies 




Table 2: Variables Used in the Study 


Background Variables 

Data Range 

Data Source 


0-1 


Gender 

(Female) 


Ethnicity 
(ALAN A or not) 

0-1 
(ALAN A) 

IR*’s fall 2007 opening enrollment 
data file (original source: final fall 
2007 Admission data file) 


0-1 

First Generation 

(V' Generation) 



Other Independent 
Variables 

Data Range 

Data Source 

H.S class rank percentile 

1-100 

IR’s fall 2007 opening enrollment 
data file (original source: final fall 
2007 Admission data file) 

SAT scores 

(Math, Critical Reading & 
Writing) 

1-800 

Number of AP credit hours 
( at entry to Ithaca College ) 

1-800 

IR’s study of FTFT AP and transfer 
credits 

H.S. academic GPA 
(4-point scale; converted if not 
4-point scale originally) 

1-4 

High school transcript review 

Strength of high school 
schedule 
(10-point scale) 

1-10 


Dependent Variable 

Data Range 

Data Source 

Cumulative Ithaca College 
GPA at end of 6th semester 

1-4 

IR’s spring 2010 course enrollment 
data file 


*IR is the Office of Institutional Research 




Statistical Models and Analysis 


Hierarchical regression is chosen as the most appropriate statistical technique to investigate 
the questions presented above. In hierarchical regression, the order of the inclusion of 
independent variables is primarily determined by a researcher which differs from other 
multivariate regression techniques such as stepwise regression. Although detailed discussions on 
statistical modeling are beyond the scope of this paper, it is useful to briefly explain how this 
statistical technique is used in this study. 

Hierarchical Regression Model 1 


Yj = pOj + pij*(Dlj) + P2j*(D2j) + P3j*(D3j) + ej 

where j=l (subscript j refers to the level of variables included in the model) 

Yj refers to the dependent variable, the 6* semester cumulative GPA at Ithaca College 
Dlj... D3J refer to dichotomous variables (ALANA, Gender and First Generation) in Model 1. 
pOj refers to the intercept of Model 1. 

pij ... P3j refer to the beta coefficients associated with predictors Dlj ... D3j. 
ej refers to the random errors of prediction for Model 1 . 

Hierarchical Regression Model 2 

Yj = pOj + pij*(Dlj) + ... + P3j*(D3j) + P4j*(Xlj) + ... + P6j*(X3j) + ej 

where j=2 (subscript j refers to the level of variables included in the model) 

Yj refers to the dependent variable, the 6th semester cumulative GPA at Ithaca College 
In addition to the independent variables included in Model 1, 

Xlj . . . X3j refer to non-SAT predictors (AP credit hours, high school GPA and Strength of 
Schedule). 

pOj refers to the intercept of Model 2. 

P4j . . . P6j refer to the beta coefficients associated with predictors Xlj . . . X3j. 
ej refers to the random errors of prediction for Model 2. 

Hierarchical Regression Model 3 

Yj = pOj + pij*(Dlj) + ... + P3j*(D3j) + P4j*(Xlj) + ... + P6j*(X3j) + 

P7j*(X4j)+... + p9j*(X6j) + ej 

where j=3 (subscript j refers to the level of variables included in the model) 

Yj refers to the dependent variable, the 6th semester cumulative GPA at Ithaca College 
In addition to the independent variables included in Model 2, 




X4j ... X6j refer to three SAT predictors (SAT Math, SAT Critical Reading and SAT Writing). 
pOj refers to the intercept of Model 3. 

pvj . . . P9j refer to the beta coefficients associated with the SAT predictors, 
ej refers to the random errors of prediction for Model 3. 

The focus of the hierarchical regression analysis is on the statistical significance associated 
with incremental change in R-square among the three models. This examines the magnitude and 
the statistical significance of the increment in the predictive validity attributed to the SAT scores 
when the predictive power associated with background variables and non-SAT evaluation 
measures is taken into consideration. 

Results: Descriptive Statistics 

Descriptive statistics are presented in Tables 3 and 4. Due to the large number of students 
missing high school class rank percentile data, it was decided to exclude this variable from the 
subsequent analyses. All predictors except for First Generation are significantly correlated with 
the 6* Semester CUM GPA at Ithaca College. Without controlling other variables, bivariate 
negative correlations of ALANA and the academic measures indicate that minority students 
appear to have lower scores in the 6* semester CUM GPA, AP credit hours and in all SAT 
scores. Female students tend to perform better at Ithaca College than their male counterparts. 
While SAT critical reading is gender neutral, male students tend to do better with SAT math 
scores and females score higher with SAT writing. It is important to note that High School 
GPA and Strength of Schedule are not significantly correlated with ALANA or First Generation 
status, revealing the importance of applying these measures to the admission process in order to 
mitigate the risk of using standardized test scores alone for college admissions. Significant 
correlations among the predictors indicate that caution is necessary because of a possible multi- 
collinearity problem in regression analysis. 



Table 3: Descriptive Analysis 


Variable 

N 

Min 

Max 

Mean 

Std. 

Dev. 

IC_6SEM_CUMGPA 

468 

2.2 

4.0 

3.35 

0.39 

ALANA 

468 

0 

1 

0.10 

0.31 

First generation college student 

468 

0 

1 

0.13 

0.34 

GENDER 

468 

0 

1 

0.59 

0.49 

High school class rank percentile 

252 

35.0 

100.0 

81.24 

14.70 

AP_CR_HRS 

468 

0.0 

48.0 

5.28 

8.57 

HS_GPA 

465 

2.0 

4.0 

3.38 

0.46 

STRENGHT_SCHEDUEE 

468 

3.0 

10.0 

7.21 

2.28 

Max SAT verbal ( includes converted ACTV) ( in 1 00s) 

468 

3.7 

8.0 

5.97 

0.81 

Max SAT math ( includes converted ACTM) ( in 100s) 

468 

3.9 

7.9 

6.00 

0.69 

Max SAT writing 

452 

3.5 

8.0 

5.83 

0.76 


Table 4; Correlations 


Variables 

IC_6SEM_ 

CUMGPA 

ALANA 

HRSTGEN 

GENDER 

AP_ 

CR_HRS 

HS_GPA 

STRENGTH_S 

CHEDULE 

SATV 

SATM 

SATW 

IC_6SEM_ 

CUMGPA 

Pearson 

1 







Sig. (2-tailed) 








N 

468 






ALANA 

Pearson 

-.118 

1 







Sig. (2-tailed) 

.010 








N 

468 

468 






HRSTGEN 

Pearson 

-.032 

.155 

1 







Sig. (2-tailed) 

.492 

.001 








N 

468 

468 

468 






GENDER 

Pearson 

.260 

.004 

.035 

1 






Sig. (2-tailed) 

.000 

.924 

.456 







N 

468 

468 

468 

468 





AP_ 

CR HRS 

Pearson 

.388 

-.121 

-.031 

.023 

1 







Sig. (2-tailed) 

.000 

.009 

.498 

.614 








N 

468 

468 

468 

468 

468 





HS_GPA 

Pearson 

.631 

-.080 

.005 

.207 

.433 

1 





Sig. (2-tailed) 

.000 

.086 

.908 

.000 

.000 






N 

465 

465 

465 

465 

465 

465 





STRENGTH, 

SCHEDULE 

Pearson 

.406 

-.041 

.053 

.119 

.489 

.490 

1 





Sig. (2-tailed) 

.000 

.380 

.255 

.010 

.000 

.000 






N 

468 

468 

468 

468 

468 

465 

468 




SATV 

Pearson 

.356 

-.116 

-.090 

.031 

.455 

.388 

.302 

1 




Sig. (2-tailed) 

.000 

.012 

.051 

.503 

.000 

.000 

.000 





N 

468 

468 

468 

468 

468 

465 

468 

468 



SATM 

Pearson 

.291 

-.146 

-.065 

-.191 

.406 

.292 

.296 

.444 

1 



Sig. (2-tailed) 

.000 

.002 

.163 

.000 

.000 

.000 

.000 

.000 




N 

468 

468 

468 

468 

468 

465 

468 

468 

468 


SATW 

Pearson 

.415 

-.132 

-.145 

.102 

.442 

.411 

.337 

.697 

.493 

1 


Sig. (2-tailed) 

.000 

.005 

.002 

.030 

.000 

.000 

.000 

.000 

.000 



N 

452 

452 

452 

452 

452 

449 

452 

452 

452 

452 


Results: Hierarchical Regression Analysis 

The results from the hierarchical regression analysis are presented in Tables 5, 6, 7 and 8. 
Table 5 summarizes the explanatory power of the overall models in a hierarchical fashion. 

Model 1 first uses only three background measures (ALANA status, gender and First Generation 
status) as predictors of the sixth semester CUM GPA in college. The result indicates 0.085 R- 
square, indicating that 8.5% of variance of the dependent variable is successfully explained by 
these three dichotomous variables. Model 2, which inserts three additional non-SAT measures 
(AP credit hours, High School GPA and Strength of Schedule) in the equation, shows that R- 
square was elevated to .437. The change in R-square attributed to these additional three non- 
SAT measures is .353, which is highly significant. Lastly, when the three SAT scores are 
inserted in the equation in Model 3, the incremental change in R-square is surprisingly small at 
.018, although the F-test on the change is still statistically significant (p=.002). This finding 
may imply that SAT scores would not predict college academic performance as effectively as 
non-SAT measures. 

To clarify this point, the R-square changes are further tested in another way by inserting 
background variables first, then three SAT scores, and lastly non-SAT measures in the equation. 
As presented in Table 6, the change in R-square attributed to three non-SAT scores after 
controlling for SAT scores is .190, ten times greater than .018, which is the change in R-square 
attributed to three SAT variables after considering the effects of non-SATs, as previously 
observed in Table 5. This affirms the finding that non-SAT measures are better than the SATs in 
predicting the sixth semester CUM GPA at Ithaca College. 

Multiple R (or correlation coefficient R) is the square root of R-square. Multiple R measures 
the degree to which a group of independent variables is correlated to the dependent variable. 



The change in Multiple R shows the increment in validity solely attributed to SAT scores in 
predicting the 6* semester CUM GPA, after the effect of the non-SAT measures as a whole is 
taken into account. As presented in Table 5, the increment of R of the SAT scores is small: 

0.012 (the difference between .669 and .657). But that incremental value is statistically 
significant. This result is similar, yet more pronounced in comparison to the 2008 College Board 
study (Kobrin et ah, 2008). The College Board study found that the incremental validity 
attributable to the SAT was 0.08 while controlling for the effect of self-reported high school 
GPA to predict the college first year GPA. The finding of this study is much smaller than the 
conclusion of the College Board study’s. We believe that this difference is due to two factors: 1) 
the College Board study used the self-reported high school GPA alone as a non-SAT predictor 
whereas Ithaca College’s study used three non-SAT variables, resulting in more predictive power 
attributed to the non-SAT measures, and 2) the College Board study used first-year GPA as a 
dependent variable whereas we used the cumulative IC GPA at the end of the 6* semester which 
is, we believe, a more stable measurement of a student’s long-term academic performance in 
college. 

Overall predictive power of Model 1, Model 2 and Model 3 is highly significant as indicated 
by the ANOVA results (Table 7). Table 8 reveals further insights about the hierarchical 
regression analysis. Collinearity statistics indicate the existence of collinearity among the 
academic predictors, but it is not severe enough to discard the analysis. In Model 1, without 
taking any other variables into account, ALANA and gender status variables appear to be 
significant predictors of academic performance at Ithaca College. In Model 2, three non-SAT 
predictors, that is, high school GPA, AP credit hours and Strength of Schedule are statistically 
significant. Two observations are noteworthy. First, as mentioned earlier “Strength of 



Schedule” measures a reviewer’s evaluation of how much a student has challenged him or 
herself in a broad array of learning in high school. The statistical significance (p<.10) of this 
predictor, unique to the present study, indicates that Strength of Schedule may quantify an 
important characteristic that cannot be evaluated by high school GPA or AP credits. Second, 
when these three non-SAT predictors are added to the equation, the ALANA status becomes no 
longer significant, which implies the importance of use of these non-SAT measures for selecting 
qualified students with minority backgrounds. 

When SAT scores are added to the model (Model 3), only SAT writing becomes significant 
in predicting students’ sixth semester performanee. This eonfirms previous findings in earlier 
studies that the SAT writing score is the best predictor of college academic performance among 
the three SAT measures (Kobrin, et ah, 2008). Gender, high school GPA and Strength of 
Schedule remain statistically significant in Model 3. The finding implies that adding only one 
SAT score — SAT writing — may marginally improve Ithaea College’s ability to prediet students’ 
performance in college. High school GPAs, Strength of Schedule and Gender remain 
statistically significant in the projeetion of students’ aeademie performanee three years after 


enrolling at Ithaca College. 



Table 5: Model Summary 


Model 

R 

R Square 

Adjusted 
R Square 

Std. Error 
of the 
Estimate 

Change Statistics 

R Square 
Change 

F Change 

dfl 

df2 

Sig.F 

Change 

1 

.291" 

.085 

.078 

.37714 

.085 

13.695 

3 

445 

.000 

2 

.661” 

.437 

.430 

.29670 

.353 

92.331 

3 

442 

.000 

3 

.675" 

.456 

.444 

.29279 

.018 

4.956 

3 

439 

.002 


a. Predictors: (Constant), FIRSTGEN, GENDER, ALANA 

b. Predictors: (Constant), FIRSTGEN, GENDER, ALANA, AP_CR_HRS, HS_GPA, STRENGHT_SCHEDULE 

c. Predictors: (Constant), FIRSTGEN, GENDER, ALANA, AP_CR_HRS, HS_GPA, STRENGHT_SCHEDULE, SATM, SATV, SATW 

d. Dependent Variable: IC_6SEM_CUMGPA 


Table 6: Model Summary 


Model 

R 

R 

Square 

Adjusted 
R Square 

Std. Error 
of the 
Estimate 

Change Statistics 

R Square 
Change 

E 

Change 

dfl 

df2 

Sig.E 

Change 

1 

.291a 

.085 

.078 

.37714 

.085 

13.695 

3 

445 

.000 

2 

.515b 

.265 

.255 

.33899 

.181 

36.258 

3 

442 

.000 

3 

.675c 

.456 

.444 

.29279 

.190 

51.162 

3 

439 

.000 


a. Predictors: (Constant), FIRSTGEN, GENDER, ALANA 

b. Predictors: (Constant), EIRSTGEN, GENDER, ALANA, SATV, SATM, SATW 

c. Predictors: (Constant), EIRSTGEN, GENDER, ALANA, SATV, SATM, SATW, AP_CR_HRS, HS_GPA, 
STRENGHT_SCHEDULE 

d. Dependent Variable: IC_6SEM_CUMGPA 


Table 7; ANOVA 


Model 

Sum of 
Squares 

df 

Mean 

Square 

F 

Sig. 

1 

Regression 

5.843 

3 

1.948 

13.695 

.000” 


Residual 

63.293 

445 

.142 




Total 

69.136 

448 




2 

Regression 

30.227 

6 

5.038 

57.229 

. 000 " 


Residual 

38.909 

442 

.088 




Total 

69.136 

448 




3 

Regression 

31.502 

9 

3.500 

40.829 

. 000 ‘^ 


Residual 

37.635 

439 

.086 




Total 

69.136 

448 





a. Dependent Variable: IC_6SEM_CUMGPA 

b. Predictors: (Constant), FIRSTGEN, GENDER, AEANA 

c. Predictors: (Constant), FIRSTGEN, GENDER, AEANA, AP_CR_HRS, HS_GPA, 
STRENGHT_SCHEDUEE 

d. Predictors: (Constant), FIRSTGEN, GENDER, AEANA, AP_CR_HRS, HS_GPA, 
STRENGHT_SCHEDUEE, SATM, SATV, SATW 


Table 8; Coefficient Analysis 



Unstandardized 

Standardized 



Correlations 

Collinearity 

Model 

B 

Std. 

Error 

Beta 

t 

Sig. 

Zero- 

order 

Partial 

Part 

Tolerance 

VIE 

1 (Constant) 

3.240 

.029 


113.402 

.000 






ALANA 

-.132 

.060 

-.101 

-2.203 

.028 

-.105 

-.104 

-.100 

.985 

1.015 

GENDER 

.215 

.036 

.270 

5.949 

.000 

.269 

.271 

.270 

.999 

1.001 

FIRSTGEN 

-.042 

.054 

-.035 

-.770 

.442 

-.038 

-.036 

-.035 

.984 

1.016 

2 (Constant) 

1.735 

.112 


15.510 

.000 






ALANA 

-.070 

.047 

-.053 

-1.473 

.141 

-.105 

-.070 

-.053 

.972 

1.029 

GENDER 

.121 

.029 

.152 

4.142 

.000 

.269 

.193 

.148 

.947 

1.056 

FIRSTGEN 

-.045 

.043 

-.038 

-1.059 

.290 

-.038 

-.050 

-.038 

.980 

1.021 

AP_CR_HRS 

.005 

.002 

.120 

2.800 

.005 

.388 

.132 

.100 

.697 

1.435 

HS_GPA 

.417 

.037 

.490 

11.322 

.000 

.623 

.474 

.404 

.681 

1.469 

STRENGTH_SCHEDULE 

.016 

.008 

.092 

2.090 

.037 

.412 

.099 

.075 

.654 

1.528 

3 (Constant) 

1.260 

.174 


7.259 

.000 






ALANA 

-.052 

.047 

-.040 

-1.099 

.272 

-.105 

-.052 

-.039 

.960 

1.041 

GENDER 

.131 

.030 

.165 

4.336 

.000 

.269 

.203 

.153 

.859 

1.165 

FIRSTGEN 

-.020 

.043 

-.017 

-.467 

.641 

-.038 

-.022 

-.016 

.955 

1.048 

AP_CR_HRS 

.003 

.002 

.063 

1.397 

.163 

.388 

.067 

.049 

.605 

1.654 

HS_GPA 

.383 

.038 

.449 

10.159 

.000 

.623 

.436 

.358 

.634 

1.576 

STRENGTH_SCHEDULE 

.013 

.008 

.074 

1.690 

.092 

.412 

.080 

.059 

.645 

1.550 

SATM 

.042 

.025 

.073 

1.649 

.100 

.294 

.078 

.058 

.633 

1.579 

SATV 

.004 

.025 

.009 

.173 

.863 

.369 

.008 

.006 

.470 

2.130 

SATW 

.058 

.027 

.113 

2.122 

.034 

.418 

.101 

.075 

.440 

2.272 


a. Dependent Variable: IC_6SEM_CUMGPA 


New Evidence from the First Test-Optional 2013 Cohort 

The above research was completed in fall 2010 whereas the Ithaca College’s test-optional 
policy was officially announced in spring 2012. The research results played a pivotal role in 


gaining institutional approval for moving Ithaca College to a test-optional policy in 2012 for 
admission of the 2013 entering cohort. 

In one of his recent publications, Ithaca College President Rochon wrote, “We expected that 
eliminating standardized tests as a required element of the application would enable us to 
increase the number of highly qualified applicants to the college, increase the quality of the 
enrolled freshman class, and increase the diversity of that class. And we fared well against those 
goals.” (Rochon, 2013). 

In fact, the College’s freshman applications increased by more than 13% in 2013. ALANA 
applications surged by more than 23% while the non-ALANA group was up by 10%. Twenty- 
eight percent of the total applicants opted out from the submission of SAT scores. 40% of 
ALANA applicants chose to opt out of the test score submission while 23% of non-ALANA 
students selected this option. A chi-square test indicates the test-optional difference between 
ALANA and non-ALANA students is highly significant. 

Furthermore, when high school GPA and class rank were used to measure the academic 
quality of the applicants, average high school GPA was slightly lower than the previous year by 
.02 point while class rank average was identical to the class of 2012. 

Building upon this robust application base, Ithaca College successfully enrolled 1789 
freshmen, 89 students more than the goal of 1700. The 2013 class is the most diverse in the 
College’s history; that is, students with minority backgrounds account for 22% of the freshman 
class in comparison to 18% of the previous year. Ithaca’s research team plans to conduct a 
follow-up study by measuring academic performance of the 2013 class who opted out of SAT 
submission in comparison to those who did not. More detailed analysis on this topic will be 


presented in the near future. 



Conclusion 


Ithaca College, a mid-sized four- year private college in central New York, successfully 
implemented a test-optional policy in 2012 for admission of the 2013 entering cohort. This study 
has discussed research methodology and results which played a pivotal role in gaining 
institutional approval for moving the College to the test-optional practice. To date, little 
research on this subject has been done by smaller comprehensive institutions like Ithaca College, 
which promotes experiential and integrative learning by combining theory and practice primarily 
for undergraduates. Such schools could be best suited for instituting a test-optional admission 
policy. This study shares useful research information with the institutions considering 
implementation of a test-optional admission policy. 

Using 468 cases which were stratified and randomly selected from the fall 2007 entering 
cohort, the study investigated the incremental validity of SAT scores in predicting the 
semester cumulative GPA in college when the effects of background variables (minority, gender, 
and first generation status) and non-SAT predictors (High School GPA, AP credits, and Strength 
of High School Schedule) were statistically taken into consideration. 

Hierarchical regression analysis was conducted, which allowed us to insert three background 
variables at first, then three non-SAT predictors, and finally three SAT scores in the equation. 

The change in R-square attributed to three non-SAT measures was 35.3%. In contrast, the 
incremental R-square change associated with three SAT scores was only 1.8% although the F- 
test on the change was statistically significant. Even with the relatively small sample, this 
finding confirms the results of previous large studies, indicating that standardized tests add 
relatively small power in predicting students’ academic performance in college. 



The present study has also revealed the critical importance of Strength of Schedule along 


with high school GPA and AP credits in the admission process if an institution does indeed 
decide to implement a test-optional policy. These non-SAT measures seem to play a particularly 
significant role in admitting qualified students from minority groups. 

By instituting a test-optional policy coupled with other strategies, Ithaca College 
successfully increased applications by more than 13% in 2013 compared to one year ago, while 
maintaining the essentially identical academic quality of applicants. ALANA applications 
surged by more than 23% and as a result, the fall 2013 freshman class is the most diverse in its 
history, with 22% of the class from minority groups. Ithaca College’s experience indicates that 
adopting a test-optional policy could be one good practice to foster diversity on campus while 
maintaining race-neutral admission policies. More research is needed to link test-optional and 
race-neutral admission policies as society intensely debates the issue of admission policies. 

Notes 

1. Approximately 850 institutions were test-optional schools in 2012, according to SAT Wars: 
The Case for Test-Optional College Admissions, edited by J. A. Soares, Teachers College, 
Columbia University, New York and London. 
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Influence of factors on cumulative GPA of freshman and transfer students 

Abstract 

Binghamton State University, a doctoral institution enrolls freshman and transfer students 
for the baccalaureate degree. This study was carried out to find if there was a difference in 
academic achievement measured in terms of cumulative GPA between freshman and transfer 
students using T-test and also to examine if the demographic, cognitive, non-cognitive and 
financial aid variables had significant influence on the cumulative GPAs of both freshman and 
transfer students using multiple regression analysis. The results revealed that senior freshman 
students scored a slightly higher cumulative GPA as compared to the senior transfer students. 

The study also provided evidence that non-cognitive factors of college impression and college 
academic experience for native students; cognitive variables of SAT score, community college 
GPA and Advanced Placement (AP) credits; demographic variables of gender and ethnicity were 
the significant variables and predictors of cumulative GPA’s of both the groups. 

Introduction 

Undergraduate students enter four year colleges and universities as freshman and transfer 
students. Freshman students start their academic career at a 4 year university immediately after 
high school and transfer students change over to a prospective 4 year university from another 
postsecondary institution to obtain their baccalaureate degree. According to the State University 
of New York NYSUNY 2020 plan, Binghamton University’s admissions office will increase 
enrollment by 500 students, which includes New Freshman and New Transfer students every 
year for the next 5 years with the goal of a total enrollment of 2,000 new students by the end of 
year 2017. The ratio of New Freshman to New Transfer students normally enrolled in 
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Binghamton University has been approximately 7 to 3 from years 2008 - 2012. Administrators 
at Binghamton University need to make a decision about whether to project the same or increase 
the ratio of freshman to transfer students based on the past academic success such as cumulative 
GPA, timely graduation of freshman and transfer students. 

Over the years, administrators raised the question related to the academic success of 
transfer students when compared to that of freshman students. Several studies examined the 
academic performance and experiences of community college transfer students when compared 
to freshman university students (Gold, 1971; Nolan & Hall, 1978). Many of these studies 
revealed that community college transfer students had lower GPAs when compared to their 
freshman student counterparts (Gold, 1971; Nolan & Hall, 1978; Hoyt & Winn, 2004). 

Beside the comparison of cumulative GPA between the two groups, the identification of 
the significant cognitive, demographic, non-cognitive and financial aid variables are important to 
predict student’s academic achievement such as cumulative GPA in the four year institution. 
Student’s ability to get higher cumulative GPA can increase graduation and retention rate. The 
present study could help administrators determine how to best prepare the incoming freshman 
and transfer students to achieve the desired outcome of timely graduation. 

Brief Review of Literature 

In this study, I investigated the influence of cognitive, non-cognitive, demographic and 
financial aid factors on cumulative GPA of both freshman and transfer students. First, I briefly 
reviewed the current literature on the cognitive variables in relation to students’ academic 
performance as mentioned in the following paragraph. 

Several researchers (e.g., Graham and Dalam, 1986; Emires and Mullen, 1997; Cejda et 
ah, 1998; Carlan and Byxbe, 2000) examined cognitive variables such as transfer GPA and 
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standardized test scores to identify students at risk for attrition (drop-out). Graham and Dallam 
(1986) found that in comparison to freshman students, transfer students were more likely to be 
placed on academic probation as a result of their lower first semester GPAs at the four year 
institute. Emires and Mullen (1997) observed that significant predictors of graduation among 
students who transferred to the University of Missouri in 1987 and 1988 were the student’s 
transfer GPA and the student being enrolled at the Engineering Campus. Carlan and Byxbe, 
(2000) also observed that the significant predictors of upper division GPA for community 
college transfers were lower division GPA and choice of major at the four year institution. 
Variables not contributing significantly were gender, associate degree attainment, full time or 
part time enrollment, transfer credit hours and majoring in the arts. 

Next, as student’s pre-entry attributes played an important role in academic performance, 
I made an attempt to research on both demographic and cognitive variables on the students’ 
performance. A group of investigators (e,g., Towsend et ah, 1993; Glass and Harrington, 2002) 
examined demographic variables such as age, gender, ethnicity, socio-economic status along 
with cognitive variables such as entering GPA as predictors of university academic success and 
degree persistence. Duggan and Pickering (2008) used logistic regression model and examined 
demographic variables, gender and ethnicity in predicting academic success/difficulty for 
freshman transfer students and the result indicated both gender and ethnicity were significant 
variables. 

As I found both demographic and cognitive variables may not sufficiently explain the 
academic performance of the students, the addition of non-cognitive factors in my study may 
explain students’ academic performance better. So I discussed literature on this issue in the 
following two paragraphs. 


3 



While considering the relationship between non-cognitive factors and student’s success, 
Tinto (2005) classified the variables into five primary categories e.g., pre-entry attributes such 
as age, gender, racial group affiliation, and marital status; student commitments such as student’s 
overall satisfaction and sense of belonging; institutional experiences such as interaction with 
staff and with faculty both inside and outside the classroom; academic and social integration 
such as classroom experience, student effort, faculty interaction, peer relationship and social 
involvement; and student outcome which is attainment of baccalaureate degree; and all these 
non-cognitive factors played significant role in a student’s success such as retention at a four 
year institution. Strauss and Volkwein (2004) observed that the various measures of academic 
and social integration and growth exerted roughly the same influence among first year students at 
both two-year and four- year institutions. 

Next, I briefly discussed the prior work accomplished on non-cognitive, cognitive and 
demographic variables in relation to student’s academic performance. 

Combining non-cognitive with cognitive and demographic variables was accurate in 
predicting academic success such as retention for student athletes as well as for traditional 
freshmen (Pickering et al., 1992). Duggan and Pickering (2008) used a number of variables on 
non-cognitive, cognitive, and demographic factors in relation to student’s academic difficulty/ 
success such as grade point average measured on a scale of 4.0 and persistence to identify the 
students who were at risk, based on the original study by Pickering et al., (1992). This study 
presented evidence that non-cognitive factors could be used to predict academic success for the 
first year transfer students. The influence of non-cognitive variables differed depending upon 
whether the person was a freshman, sophomore and upper division (junior/senior) transfer 
student. 


4 



Finally, I wanted to include financial aid in my model and I briefly reviewed on this 
aspect also. Strauss and Volkwein (2002) examined financial aid and attitudes among the other 
variables and the result indicated that students who faced financial constraints, and received 
financial aid in the form of federal and state grants had higher institutional commitments leading 
to better academic performance than those who did not receive financial aid under similar 
circumstance. In the study, financial aid, though not strong, was found to be significant variable 
on predicting institutional commitment of a two year and four year institutions. 

While a great deal of research existed regarding the impact of above mentioned cognitive, 
non-cognitive, demographic and financial factors on the academic performance such as retention 
and graduation rate of all students at four year institutions, little had been done to study the 
impact of those factors on the relative academic achievement such as cumulative GPA on both 
the groups of freshman and transfer students separately at a 4 year institution. The goal for my 
study was to examine whether cognitive, non-cognitive, demographic and financial aid factors 
could be related to cumulative GPA of both freshman and transfer students at Binghamton 
University, thereby identifying those students who are at risk. 

The purpose of my study was to compare overall cumulative GPA of freshman students 
with that of transfer students. Additionally, the study compared the significant predictors based 
on their demographic, financial, cognitive and non-cognitive factors that influenced the 
cumulative GPAs of freshman and transfer students. My research questions were: 

1) Does a significant difference exist between freshman and transfer students’ academic 
achievement as measured by Grade Point Averages GPA (ranges between 0-4 scales) for 
all graded courses taken at the university using Parametric T-test? 
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2) Do cognitive (e.g., SAT scores, prior institution transfer GPA), demographic (e.g., 
gender, ethnicity), financial aid (e.g., received financial aid or not) and non-cognitive 
factors assigned as factor score using factor analysis on the likert scale Student Opinion 
Survey (eollege impression, aeademic experienee and environment, sense of eommunity 
and career services) predict the cumulative GPA of freshman and transfer students using 
multiple regression analysis? 

Method (Design) 

Instrumentation 

The American College Testing (ACT) is an independent non-profit organization 
providing assessment, research information and program management services to the education 
and workforce development industries. The Student Opinion Survey (SOS) was an instrument 
designed by ACT to explore enrolled student’s satisfaction with programs, services and other 
aspects of their college experience http://www.act.org . Binghamton University administered the 
SOS survey in April 2006 to assess student’s satisfaction and the college experience at 
Binghamton University. Reliability and Validity information of SOS survey was given in the 
http : //apps . airweb . or g/s urve ys/measureq ualit y .pdf . As mentioned in this guide, the assessment 
instrument had a respectable level of reliability, but reliability was much dependent how well the 
sample was randomly selected from the population. 

I selected a few items (as shown in Appendix A) from SOS survey. Students’ responses 
(measured on a likert scale) collected from the SOS survey were further matched with their 
demographic, financial aid and cognitive information extracted from students university records. 

The independent/predictor variables consisted of non-cognitive variables, which were 
responses based on selected items in the areas of college impressions, academic environment & 
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facilities, sense of community, student life, career services and eollege environment from SOS 
survey (as shown in Appendix A) and eonsisted of demographic, financial and cognitive 
variables such as age gender, tuition assistanee program offer ete., which were collected from 
student’s university reeords (as shown in Appendix B). 

The dependent/outeome variable was overall eumulative GPA a student attained at the 
time of the SOS survey eonducted in Binghamton University. 

Subjects/Participants- 

The target population eonsisted of 10,870 degree- seeking undergraduate students, who 
were enrolled as freshman, sophomore, junior or senior level during Spring 2006. 

A stratified random sampling was used to seleet sample of 3,590 students and the SOS 
survey was sent to this identified sample. Out of the identified sample 3,590 students, 1,228 
students responded resulting in a response rate of 34%. The dataset of 1,228 reeords was further 
matched with their demographic, financial aid and cognitive information of student university 
reeords. Some of the students had missing unique identifier on the survey responses, so I was 
unable to mateh survey responses of the unique identifiers with the identifier found on the 
student university reeords and so I deleted the missing identifier reeords. The final dataset 
eonsisted of 1206 reeords, a response rate of 33.5%. 

Procedures and Planned Analysis 

I examined the descriptive statisties, frequency and sample size for all categorieal 
variables such as gender, ethnicity etc. and examined the mean and sample size for eaeh metric 
variables sueh as SAT scores, prior institution transfer GPA, etc. (as shown in Table 1), which 
were further sub-grouped as “Entered as Freshman” and “Entered as Transfer”. Based on my 
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first hypothesis question, I performed a t-test analysis to examine if there was a significant 
difference in cumulative GPA between freshman and transfer students on academic achievement. 
I set level of significance at 0.05. As shown in Table 2, mean, standard deviation and p-values 
were reported for cumulative GPAs of the two groups “Freshman” and “Transfer” students. The 
difference in cumulative GPA between freshman and transfer was 0.02, difference between the 
two scores was insignificant with a p-value was < 0.57. 

Based on second hypothesis question, I performed a multiple regression analysis to 
examine if any of the independent variables consisting of demographic, cognitive, non-cognitive 
and financial variables had statistical influence on the cumulative GPA of freshman and transfer 
students. In the first stage of second hypothesis question, I used multi-collinearity test to identify 
the highly correlated variables among the demographic, financial aid and cognitive variables 
because the presence of highly correlated independent variables in the model might lead to 
inconclusive results. In the second stage, factor analysis was performed using principal 
component with varimax rotation to select the factors with eigenvalues of at least 1.0 and 
inclusion of items to have loading of at least 0.40. In my study, the final three factors were 
identified as college impression and academic experience, college environment and college 
services. Factor score was assigned to each score. Factor score was obtained by summing raw 
scores corresponding to all items having loading at least of 0.40 for each factor. The goal of 
using varimax rotation was to minimize the complexity of the components by making the large 
loadings larger and the small loadings smaller within each component. In this study, the final 
three factors were included as non-cognitive variables in my multiple regression models 1 & 2. 
Finally, stepwise selection was examined for two multiple regression models, one each model for 
Freshman and Transfer students. Stepwise selection was preferred when the number of predictor 
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variables in the regression models increased and also when multicollinearity was present among 
the predictor variables (http://en.wikipedia.org/wiki/Stepwise_regression). In stepwise selection, 
the final model was achieved when no predictor variables outside the model met SLE 
(significance level to enter) = 0.15 and all other predictor variables passed the SLS (significance 
level of stay) =0.15. The default values of SLS and SLE are 0.15. Table 3 depicted the regression 
unstandardized, standardized coefficients and standard errors for each significant predictors for 
predicting undergraduate cumulative GPA of a student for the models; Model 1 showed students 
“Entered as Ereshman” and Model 2 showed students “Entered as Transfer”. I set the 
significance level at 0.05. Based on the multicollinearity test and criterion used in the factor 
analysis, fewer variables were selected in the multiple regression models (as shown in TableS). 
R-square value increased as the number of variables increased in the model, so I considered 
adjusted R- square value to summarize the overall fitness of the model. R-squared is defined as 
the percentage of variability of the outcome variable cumulative GPA, that was explained by the 
predictors in models. 

Results and Analysis 

Table 1 here 

As shown in Table 1, freshman (N= 917) and transfer (N = 289), the percentage of 
nominal variables such as gender, fulltime/part-time, in-state/out-of-state and ethnicity were 
almost the same for freshman and transfer respondents. The percentage of females for freshman 
and transfer were 58.5% and 62.4% respectively and were higher than the males in both the 
groups. The full-time students for freshman and transfer were 100% and 100% respectively and 
the respondents were mostly from in-state with 91.3% freshman and 92.7% transfer respondents. 
The largest percentage of respondents was white 54%-59% approximately followed by Asian 
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12.4%-15.9% and then Black 3.1%-3.9% for freshman and transfer respondents. All the 
respondents were degree- seeking in both freshman and transfer category. The largest percentage 
of freshman respondents entered Binghamton University with student status as “Freshman” and 
“Sophomore” but largest percentage of transfer respondents entered Binghamton University with 
student status as “Sophomore” and “Junior” based on the credit hours transferred by the prior 
institution or attained at Binghamton University. Approximately, 56%-57% of the respondents 
received financial aid in both the groups. Mean age of transfer students were slightly greater by 
2.3 years than the freshman students. Freshman students had a mean SAT combined score of 
1239 measured on a total score of 1600 point section (Mathematics and Verbal), and transfer 
students had a mean 3.30 GPA measured on a scale of 4.0 from prior institution. 

Table 2 here & Table 3 here 

Based on research question 1, the difference in cumulative GPA between freshman 
(M=3.19, SD=0.45) and transfer (M=3.17, SD=0.51) students was 0.02. Testing the equality of 
variances, F (285,914) = 1.13, p-value=0.18, indicating that the variances were equal in both the 
groups of freshman and transfer students. Considering that the variances were equal, the 
differences in cumulative GPA between freshman and transfer respondents was statistically 
insignificant, t(l 199) = 0.57, p-value 0.58. But if we look at the tabled, there is a significant 
difference between in scores between the freshman seniors and transfer seniors. 

Table 4 here 

Based on research question 2, the final multiple regression Model 1 (students entered as 
Freshman) and Model 2 (students entered as Transfer) were shown in Table 3, which displayed 
unstandardized coefficients (B), standardized regression coefficients ( P), standard error of 
unstandardized coefficients (SE (B)) and adjusted R-squared. The models for predicting 
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cumulative GPA for freshman, F (9,877) =38.96, p < 0.0001 and transfer, F (8,203) = 6.99 and p 
<0.0001 were statistically significant. The adjusted R-squared for Model 1 (Entered as 
Freshman) was 0.27, indicating that 27% of the variance of the outcome variable cumulative 
GPA ean be aeeounted by the predietors in the model 1. The adjusted R-squared for Model 2 
(Entered as Transfer) was 0.18, indicating that 18% of the variance of the cumulative GPA could 
be aeeounted by the predietors in model 2. The unstandardized regression coefficients described 
the strength and direction of the relationship between each predictor and the outcome variable 
eumulative GPA. Standardized regression coeffieients would determine the relative influenee of 
each predictor on the outcome variable cumulative GPA. 

Demographic Variables- The demographic variables such as gender, age, Hispanie, senior were 
significant predictors in Model 1 at 0.05 level. The unstandardized coefficient for male is -0.19, 
indieating that, on average, cumulative GPA of male was 0.19 lower than that of female after 
controlling the other variables in the model 1 . Similarly, the unstandardized coefficient for age 
was -0.03, indicating that increase of one year of age, cumulative GPA decreased by 0.03. 

The standardized regression coefficient for age was -0.07, gender was -0.20. 

In model 2, none of the demographic variables are signifieant. 

Cognitive variables- In model 1, variables AP credits and SATC score were significant 
predictors and in model 2, variables AP credits and Transfer GPA (from prior institution) were 
significant predictors. The variable transfer GPA had the highest influence on cumulative GPA 
in model 2, indicating that transfer students with higher prior institution GPA had probability of 
getting higher cumulative GPA in a 4 year university. 

Non-Cognitive variables- In the models 1, college impression & academic experience factor 
were significant. In model2, none of the non-cognitive variables were signifieant. 
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Financial Aid- None of the financial aid variables was significant in model land model 2. 

Discussion and Conclusion 

Based on hypothesis question 1, overall there was no statistically significant difference 
in cumulative GPA between freshman and transfer students, but comparing cumulative GPA at 
student level, freshman seniors scored 0.14 higher than senior transfer students and the 
difference was statistically significant. So the question was if the difference was meaningfully 
significant? To answer this question, we need to know the diverse culture and social background 
of transfer students. These students were more likely to attend a community college because of 
low tuition costs and less competition than to going to a 4 year institution (Sara Goldrick-Rab et 
ah, 2009). Alfred Herrera, assistant vice provost at the University of California explained that 
one of the goals of higher education was to prepare students to compete and perform well in the 
real world. He also added that welcoming students from a variety of areas- geographic, cultural, 
ethnic, age was a critical factor in that preparation (Handel, 201 1). So supporting the process of 
admission and enrollment of transfer students at Binghamton University or any other 4 year 
institution should be one of the primarily objectives for giving an equal opportunity to any 
student coming from a diverse background to get their baccalaureate degree. 

My study presented evidence that college impression & academic experience, which 
involved overall impression of the quality of education at this college, satisfaction level at this 
college in general, meeting faculty inside and outside the classroom etc., was the most significant 
variable for freshman students. The results of my study were consistent with the findings of 
Tinto (2005) in arguing that institutional experiences, academic integration and outcome played 
a significant role in a student’s success at a four year institution. Administrators should consider 
a range of success strategies like developing high quality teaching and instruction, providing 
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quality vocational and pre-requisites programs, ereating positive eollege environment, mentoring 
and tutoring serviees etc. to prepare the freshman and transfer students to aehieve the desired 
outcome of timely graduation and better cumulative GPA at Binghamton University. 

Cognitive variables sueh as SAT seores, AP eredits and prior institution transfer GPA, 
demographie variables sueh as gender, ethnieity, deelaring their major or not and student level 
were the signifieant eognitive and demographie indieators, whieh inerease the ability to prediet 
students cumulative GPA. Townsend et el. (1993) also examined demographic and cognitive 
variables and found that eommunity eollege GPA was the signifieant predietor of university 
academie sueeess and degree persistenee whieh is eonsistent with the findings of my study. It is 
found that students deelaring their majors and as well having senior status seores slightly better 
than freshman, sophomore and juniors. 

The limitation of the study was that the survey was eondueted during the spring 2006 
and so the behavior and pereeption of freshman and transfer students towards their aeademie 
experience and environment might have changed during the last few years and so the latest data 
of Student Opinion Survey might be eonsidered for identifying the key signifieant indicators for 
higher eumulative GPA. The sample size of transfer students was less as eompared to freshman 
and so in the future this study should be tested with larger sample size for both freshman and 
transfer students. My study findings should also be tested with other similar kinds of setting sueh 
as publie and researeh 4 year institutions. 

Serviees sueh as tutoring and mentoring have been started at our doetoral institution to 
help and mentor the aeademieally weak students. Identifying the signifieant variables influeneing 
overall eumulative GPA for freshman and transfer students ean help us to eome up with more 
effeetive strategies to prepare the ongoing ineoming freshman and transfer students in the future. 
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Table 1 Descriptive Statistics of Cognitive, Demographic and Financial Aid Variables 


Variable 

Variable Type 

Statistics 

Freshman 

Transfer 

Gender 

Nominal Variable 

N(%) 

Male = 380-(41.5%) 
Female = 537(58.5%) 

Male= 109 (37.6%) 
Female = 180(62.4%) 

FTPTIND 

Nominal Variable 

N(%) 

Fulltime= 917(100%) 

Fulltime= 289 (100%) 

In_state 

Nominal Variable 

N(%) 

Instate=837 (91.3%) 
Outstate= 80(8.7%) 

Instate=268 (93.1%) 
Outstate=2 1(6.9%) 

Ethnicity 

Nominal Variable 

N(%) 

American 

Indian=l(0.1%) 

Asian = 151(16.4%) 
Black = 38(4.1%) 
Hispanic = 45(4.9%) 
NRA =8(0.9%) 
Unknown= 175(19.1%) 
White =499(54.4%) 

American Indian= 
0(0%) 

Asian = 38(13.5%) 
Black =9(3.1%) 
Hispanic =9 (3.1%) 
NRA = 5(1.7%) 
Unknown = 57(19.7%) 
White = 171(59.2%) 

Stud_level 

Nominal Variable 

N(%) 

Freshman=92(10.0%) 

Sophomore=253(27.6%) 

Junior=217(23.7%) 

Senior=355(38.7%) 

Freshman=20(6.9%) 
Sophomore=4 1 (14.2%) 
Junior=79(27.3%) 
Senior=149(51.6%) 

Deg_seek 

Nominal Variable 

N(%) 

Degree seeking= 917 
(100%) 

Degree 

seeking=289( 100%) 

Offcamp 

Nominal Variable 

N(%) 

Offcampus=27 6(30. 1 %) 
Incampus=641(69.9%) 

Offcampus=153(52.9%) 
Incampus= 136(47.1%) 

rcvdaid 

Nominal Variable 

N(%) 

Yes-526 (57.4%) 
No-391(42.6%) 

Yes- 164(56.7%) 
No- 125(43.1%) 

Tapofferl 

Nominal Variable 

N(%) 

Yes-455(49.6%) 

No-462(50.4%) 

Yes- 149(5 1.6%) 
No- 140 (48.4%) 

Ap_credits 

Nominal Variable 

N(%) 

Yes-599(65.3%) 
No-3 18(34.7%) 

Yes-56(19.4%) 

No-233(80.6%) 

Ugcumhrs 

Metric Variable 

N 

Mean 

N=917 

Mean=74.0 

N=289 

Mean=85.4 


16 


ugcumgpa 

Metric Variable 

N 

Mean 

N=915 

Mean=3.19 

N=286 
Mean= 3.17 

aidacpt 

Metric Variable 

N 

Mean 

N=917 

Mean=7165 

N=289 

Mean=8113 

Age 

Metric Variable 

N 

Mean= 

N=917 

Mean=20 

N=289 

Mean=23 

SATC 

Metric Variable 

N 

Mean 

N=889 

Mean=1239 

N=109 

Mean=1178 

Transgpa 

Metric Variable 

N 

Mean 

NA 

N=218 

Mean=3.3 

Transfer 
credit hrs 

Metric Variable 

N 

Mean 

N=917 

Mean=5.3 

N=289 

Mean=50.4 
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Table2. Comparison of cognitive factors between Freshman and Transfer students 


Cognitive Variable 

Freshman 

Transfer 

Difference 

p-value 


Mean 

SD 

Mean 

SD 



1) Cum. GPA 

3.19 

0.48 

3.17 

0.51 

0.02 

0.57 

p-value <0.05 *, SD = standard deviation 





Tables. Comparison of cognitive factors between Freshman and Transfer students by student 
level 

Cognitive Variable by 


Freshman 

Transfer 

Difference 

p-value 

Student Level 








Mean 

SD 

Mean 

SD 



Cum. GPA (Freshman) 

2.83 

0.54 

3.01 

0.78 

0.18 

0.36 

Cum. GPA (Sophomore) 

3.08 

0.53 

3.11 

0.42 

0.03 

0.75 

Cum. GPA (Junior) 

3.21 

0.42 

3.18 

0.56 

0.03 

0.64 

Cum. GPA (Senior) 

3.34 

0.38 

3.2 

0.46 

0.14 

0.001* 
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Table4 Multiple Regression Models for Freshman and Transfer students for the outcome 
dependent variable cumulative GPA 



Freshman Students (Model 1) 

Transfer Students (Model 2) 

Variables 

B 

SE(B) 

P 

B 

SE(B) 

P 

Intercept 

2.01 

0.39 


0.12 

0.42 


Male 

-0.19** 

0.03 

-0.20 

- 

- 

- 

Senior 

0.17** 

0.04 

0.12 

- 

- 

- 

Declared_maj or 

0.17** 

0.04 

0.15 

- 

- 

- 

Age 

-0.03 

0.02 

-0.07 

0.01 

0.006 

0.11 

Hispanic 

-0.26** 

0.06 

-0.01 

-0.32 

0.20 

-0.1 

Ap_credits 

0.15** 

0.03 

0.15 

0.27* 

0.09 

0.12 

Sate 

0.001** 

0.00011 

0.27 

- 

- 

- 

Transgpa 

- 

- 

- 

0.46** 

0.09 

0.34 

College 
impression & 
acad experience 

0.02** 

0.003 

0.21 

0.008 

0.005 

0.09 

Adjusted R- 
squared 

0.28 



0.19 




Note: B = unstandardized regression coeffieient estimates, SE B= standard error of 
unstandardized regression coefficient, P = standardized regression coefficient, R-squared = 
Coefficient of Determination; *p < 0.05. **p < 0.01 
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Appendix A Description of Predictor (Non-Cognitive) Variables included in the 
Regression Models 


Variable 

Survey Question 

Coding Scheme(Likert Scale) 

College Impressions 

Plantograd 

Is your current plan to 
graduate from this college? 

5= Definitely yes 
4= Probably yes 
3= Uncertain 
2= Probably no 
1= Definitely no 

Qual_edu 

What is your overall impression 
of the quality of education at this 
college? 

5= Vey High 
4=High 
3= Average 
2= Low 
1= Very Low 

General 

How satisfied are you with 
this college in general? 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Academic environment, experiences and facilities 

Avl_intem 

Availability of internships 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Satacad 

How frequently have you been 
satisfied with your academic 
experiences at this college? 

5= Very frequently 
4= Frequently 
3=Sometime 
2= Rarely 
1= Never 

Meetfacout 

How frequently have you had 
discussions, meetings, or 
conversations with instructors 
outside of class? 

5= Very frequently 
4= Frequently 
3=Sometime 
2= Rarely 
1= Never 

Groupwork 

How frequently have you 
worked with other students on 
class assignments? 

5= Very frequently 
4= Frequently 
3=Sometime 
2= Rarely 
1= Never 

disruptlm 

How frequently have you 
experienced classroom 
behavior by other students that 
was disruptive to learning? 

5= Very frequently 
4= Frequently 
3=Sometime 
2= Rarely 
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1= Never 

Sense of Community 

Belong 

Your sense of belonging to 
campus 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Noharass 

Freedom of harassment on 
campus 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Facrespt 

Faculty respect for students 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Rules 

Clarity of rules for student 
conduct 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Student Life 

subsabus 

Educational programs 
regarding alcohol and 
substance abuse 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Aslt_pgms 

Sexual assault prevention 
programs 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

orient 

New student orientation 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Career Services 
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Career_srv 

Career planning services 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Job_srv 

Job placement services 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

Soc_support 

Your social support network 
on campus 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

arts 

Cultural programs 

5= Very satisfied 
4= Satisfied 
3= Neither Satisfied 
nor dissatisfied 

2= Dissatisfied 
1= Very dissatisfied 

College Environment 

Prjudc_act 

Acts of prejudice are rare on 
this campus 

5= Strongly agree 
4= Agree 

3= Neither agree nor 
disagree 

2= Disagree 
1= Strongly disagree 

mentor 

I have developed a mentoring 
relationship with a 
faculty/staff member 

5= Strongly agree 
4= Agree 

3= Neither agree nor 
disagree 

2= Disagree 
1= Strongly disagree 

Advise_avl 

Academic advising is 
available to me when I need it 

5= Strongly agree 
4= Agree 

3= Neither agree nor 
disagree 

2= Disagree 
1= Strongly disagree 

openness 

I have developed an openness 
to the opinion of others 

5= Strongly agree 
4= Agree 

3= Neither agree nor 
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disagree 

2= Disagree 
1= Strongly disagree 

goals 

The college has helped me 
meet the goals I came here to 
achieve 

5= Strongly agree 
4= Agree 

3= Neither agree nor 
disagree 

2= Disagree 
1= Strongly disagree 

value 

Comparing the cost to the 
quality of education , this 
college is a good value 

5= Strongly agree 
4= Agree 

3= Neither agree nor 
disagree 

2= Disagree 
1= Strongly disagree 

Tuff_fin 

It has been difficult to finance 
my education 

5= Strongly agree 
4= Agree 

3= Neither agree nor 
disagree 

2= Disagree 
1= Strongly disagree 
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Appendix B Description of Predictor Variables ( Cognitive, Demographic and Financial 
Aid Included in the Regression Models 


Variable 

Deseription 

Coding Scheme 

Gender 

Sex of a student 

If sex= M then male=I or else 
male=0; 

Age 

Age of a student 

It ranges between 18-55 years 

FTPTIND 

Student enrolled as Fulltime or Part-time 

If FTPTIND= F then fulltime=l 
or else fulltime=0; 

In_state 

If a student is from In-state or Out-state 

If student is from instate then 
In_state=I or else in_state= 0; 

Ethnieity 

Ethnicity of a student 

If ethnicity = “White” then 
white=I or else 0; 

If ethnicity=” Asian” then 
Asian=I or elseO; 

If ethnicity=”black” then 
black=l or else 0; 

If ethnicity=”Hispanic” then 
Hispanic =I or else 0; 

If ethnicity=” American Indian” 
then amind= I or else 0; 

If ethnicity= “Non-Resident 
Alien” then nra=l or else 0; 

If ethnicity= “Unknown” then 
unknown=l or else 0; 

Underrep 

If a student’s ethnicity is “Black”, 
“American Indian” or “Hispanic” then 
student is classified as underrepresentative 

If a student is underrep then 
underrep= 1 or else 0 

Stud_level 

Student is given stud_level status based 
on their undergraduate credit hours for the 
current semester 

If stud_level in “Freshman” then 
freshman= I or else 0; 

If stud_level in “Sophomore” 
then sophomore=I or else 0; 

If stud_level in “Junior” then 
junior=I or else junior=0; 

If stud_level in “Senior” then 
junior=l or else junior=0; 

Deg_seek 

If a student is degree seeking or not 

If a student is degree-seeking 
then deg_seek= I or else 0; 

Offcamp 

If a student lives in offcampus or not 

If a student is in offcampus then 
offcampus=I or else 0; 

revd 

If a student did receive financial aid or not 

If student received financial aid 
then rcvd=l or else 0; 

aidacpt 

The aid amount accepted by the student 

It ranges between 0-35,206 $ 
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ugcumhrs 

Cumulative hours obtained by the student 

It ranges between 0-218 hours 

Ap_credits 

If a student has AP eredits or not 

If student reeeived AP eredits 
then ap_eredits=l or else 0; 

sate 

SAT(Math) + SAT(Verbal) = 
SAT(Combined) 

It ranges between 710 to 1540 

transgpa 

Transfer GPA transferred by the transfer 
student from prior institution 

It ranges between 2.14 - 4.0 seale 

ugtrnhrs 

Transfer eredit hours transferred by the 
transfer student 

It ranges between 0 to 103 eredit 
hours 

ugcumgpa 

Undergraduate eumulative GPA based on 
the last semester the student attended 
Binghamton University 

It ranges between 1.42 - 4.0 
seale. 

grad 

If a student graduated or not 

If student graduated then grad=I 
or else grad=0; 

firstchoice 

If a student’s preferenee to Binghamton 
was first ehoiee or not 

firstehoiee= 1 or else 0; 

Tapofferl 

If a student is granted Tuition Assistanee 
Program 

If a student reeeives tapofferl 
then tapoffer =I or else 0; 
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Introduction 

At the turn of the twenty-first century, the Oakland Athletics revolutionized the way baseball 
players were evaluated by moving away from conventional measures (e.g., batting average, runs batted 
in, and home runs) to measures that would increase a team's chances of winning (e.g., on-base 
percentage and slugging percentage). By understanding the rules (the team with the most runs wins), 
the Oakland Athletics used data effectively to win games in a more efficient manner (i.e., spending less 
money). This Moneyball (Lewis, 2003) approach allowed Oakland to compete against more expensive 
teams like the New York Yankees and Boston Red Sox, even though their payroll was remarkably 
smaller. 

With public funding decreasing for higher education and an increasing demand for 
accountability with regards to degree completion, institutions need to become more efficient and 
effective in graduating students. Most institutions already collect and report institutional data to the 
federal government in order to be eligible for federal monies (e.g.. Integrated Postsecondary Education 
Data System [IPEDS]). Institutions also collect and track data to ensure records are properly and 
accurately maintained for their day-to-day operations. The issue is not whether institutions collect 
enough data, but instead whether institutions are using already collected data effectively and efficiently, 
especially with respect to graduation. 

For example, institutions must report one-year retention rates to IPEDS and theoretically having 
high retention rates should lead to high graduation rates. Yet, just reporting retention rates does not 
capture if students who are being retained actually have a chance to graduate or if they are being 
retained but really have no chance of being eligible to graduate (i.e., students who are retained from 
one year to another but have a grade-point-average (GPA) less than one). As the cost of higher 
education continues to increase, retaining students who have little to no chance of earning a credential 


while accumulating a large sum of debt becomes troublesome. 
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By combining data already collected, institutions can more accurately predict graduation rates. 
Examining metrics such as retention and GPA, institutions can develop better measures (i.e., retention 
to graduation rate - which examines the retention rates by first semester GPA) to do so. Understanding 
these breakdowns allows administrators and faculty members to develop better policies and programs 
that are more effective. For example, if a large number of students are not being retained because of 
low GPA, administrators and faculty members may examine what courses are contributing to their poor 
academic performance. The purpose of this research, therefore, is to develop a model for degree 
completion from an institutional perspective. By having such a framework, institutional researchers and 
administrators may be better able to develop and evaluate measures, policies, and programs that can 
improve the graduation rates at their institutions. 

Literature review 

The field of higher education has been guided by many influential frameworks such as input- 
environment-outcomes models including Terenzini and Reason's (2005) Comprehensive Model of 
Influences on Student Learning and Persistence and sociological models such as Tinto's (1993) 
Longitudinal Model of Institutional Departure. This allows prominent and important concepts such as 
academic and social integration (Tinto, 1993) or student engagement (Kuh, Kinzie, Schuh, Whitt, & 
Associates, 2003) to guide an institution's assessment practices and policies, especially with regards to 
learning outcomes. Through their review of higher education research, Pascarella and Terenzini (2005) 
found social and academic involvement are positive influences on student persistence, but the "findings 
are inconsistent and the causal linkages remain obscure" (p. 440). One possible explanation for these 
inconsistent and causal linkages is that institutions rarely deny students their degree because of their 
involvement or engagement levels. If students meet certain academic requirements, the institution will 
more than likely award them a degree regardless of their level of involvement with professors (e.g.. 
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conducting undergraduate research) or extracurricular activities (e.g., participating in student 
organizations). 

Instead of examining retention and graduation from a student perspective, we propose an 
institutionally focused model based upon the requirements that students need to satisfy in order to 
graduate from the university. At most higher education institutions, these graduation requirements are: 
1) achieve a certain cumulative GPA and accumulate a certain number of credits in a specified 
curriculum, 2) pay tuition, and 3) do not commit any acts of extreme social or academic deviance (e.g., 
selling drugs, assaulting classmates, plagiarism). This model, thus, allows institutional researchers to 
easily operationalize variables with data already collected by the registrars and student aid offices. More 
importantly, the results can inform administrators of tangible actions that they can more easily act 
upon. Administrators, for example, can review and assess a fee policy to examine whether it is placing 
undue hardships for those least able to afford them (e.g., adding a convenience fee for those paying 
their tuition by credit card) and then change it. 

Data Sources and Methodology 

For this study, we examined longitudinal student records and financial data for first-time, full- 
time, baccalaureate-seeking students who started in the summer or fall 2004 (N = 12,212 students) at a 
large public Mid-Atlantic Research I University and its regional campuses. The student record data (first- 
year cumulative GPA, initial campus, last semester of attendance, graduation indicator) for the 2004 
cohort was obtained through the institution's data warehouse. The financial information (total federal 
aid^ total state aid, total institutional aid^ total private aid^ total aid^, cost of attendance, and income^) 
was provided by the institution's Office of Student Aid. Using this information, we derived the net cost 


* This includes veteran’s benefits. 

^ This includes University scholarships, University grants and University fellowships. 

^ This includes private loans, external scholarships, and loans 
Total aid is the sum of total federal aid, total state aid, total institutional aid, and total private aid. 
^ Income calculated by the federal processer and based on FAFSA. 
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of attendance (NCOA) by calculating the difference between the cost of attendance and total aid 
awarded. The financial aid and family income information was available only for students who 
completed the Free Application for Federal Student Aid (FAFSA). The student record information was 
retrieved for all semesters starting from summer 2004 to spring 2010 (a six-year timeframe), while 
financial information was collected based on an academic year timeframe from 2004-05 to 2009-10. 

To operationalize the proposed model, these analyses focused on academic performance, family 
income, and NCOA (i.e., cost of attendance minus financial aid received) on graduation because as long 
as a student meets the academic guidelines of a major (i.e., meets a certain GPA requirement and 
obtains the required number of credits in a prescribed curriculum) and pays tuition, she/he will 
generally earn a degree from her/his institution. In developing this model, our focus shifts from 
examining the influence of financial aid, which includes loans, grants, and work-study, to examining the 
influence of NCOA on graduation. Financial aid is important because it lowers the NCOA, however, if the 
student cannot afford to pay the NCOA, he or she cannot be enrolled. Due to the sensitivity and 
accessibility of whether a student was dismissed because of social and academic deviance, this factor 
was omitted from these analyses. We do recognize that this does occur but it is generally the exception 
and not the rule. 

For this study, we defined academic performance as the cumulative GPA at the end of the first 
academic year. In the 2004 cohort, 107 students did not have a cumulative GPA at the end of their first 
academic year. These students are included in the descriptive tables, but omitted from the logistic and 
ordinary least-squares (OLS) regression analyses. A student's family income level was determined by the 
median of the supplied income values obtained from the Free Application for Federal Student Aid 
(FAFSA) applications during the six-year timeframe. Only 16.6 percent or 2,025 students of the 2004 
cohort did not file the FAFSA at least once during the examined time period. For the descriptive tables 
supplied in the report, family income was binned according to the 2004 quintiles set by federal 
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guidelines (Tax Policy Center, 2011); however, for the logistic and OLS regression analyses, the family 
income variable was kept continuous (units = per ten thousand dollars). To prevent the exclusion of the 
students whose family income was unknown from logistic and OLS regression analyses, a dichotomous 
variable was created to indicate whether the student completed the form at least once (1 = filed form at 
least once; 0 = never filed a FAFSA form). Another variable was then created by calculating the product 
of the dichotomous variable and the family income variable. Models were then created utilizing the 
dichotomous variable and the modified family income variable to gauge the influence of family income 
on graduation and cumulative GPA without having to omit students from the analyses. 

We utilized descriptive statistics (e.g., frequencies, rates), logistic regression, and OLS regression 
to examine the relationships between academic performance, family income, NCOA for the first year, 
and six-year graduation rates. For the logistic regression models, we modeled the influence of academic 
performance, family income, and NCOA on whether students graduate or not. The OLS regressions 
modeled the influence of family income and NCOA for the first year on academic performance. 

We created separate logistic and OLS regression models for each of the family income quintiles® 
because we hypothesized that the NCOA might have differing effects at various family income levels. For 
example, a NCOA of $10,000 for a family with a combined income of $18,000 could have a more severe 
consequence than for a family with a combined income of $88,000. Due to the limitations of the data 
and the varying residential statuses of the campuses, which have a subsequent influence on cost of 
attendance (i.e., students who reside on campus have a higher cost of attendance compared to students 
who commute^), the analyses were further disaggregated in the following fashion: 1) Flagship Campus 
(the majority of first-year students are required to reside on campus), 2) Residential Regional Campuses 

® For the sake of readability, this report uses the following labels: lowest quintile (less than $18, 486), second-lowest 
quintile ($18,487 to $34, 675), middle quintile ($34,676 to $55, 230), second-highest quintile ($55,231 to $88,002), 
and highest quintile (more than $88, 002). 

^ A limitation of the NCOA variable was that it did not include the costs incurred by commuter students living off- 
campus (e.g., rent). 
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(students have the option to reside on campus during their first year), and 3) Nonresidential Regional 
Campuses (all students commute to campus). 

The strength of this study is that it utilizes population data to confirm the validity of the 
proposed model; yet, this is also a weakness as the generalizability of the results are limited to a single 
institution/system. More research needs to be conducted in order to examine whether the results are 
generalizable to other types of institutions (e.g., two-year institutions, private, for-profit). 

Results 

Table 1 provides the six-year graduation rates for all 2004 first-time, full-time, baccalaureate- 
seeking students who started in the summer or fall 2004 disaggregated by family income and 2004-2005 
cumulative GPA. Regardless of where a student starts at the University, students who performed better 
academically at the end of the first-year were more likely to graduate than those who perform poorly. 
The data also suggested that students from families with higher incomes were more likely to graduate 
than those students from poorer families. The logistic regression models (Table 2) found that for every 
one point increase in the cumulative GPA, a student increased her/his odds of graduating within six 
years by 4.49. In other words, a student with 3.0 cumulative GPA at the end of her/his first academic 
year had a 349 percent higher chance of graduating than a student with a 2.0 cumulative GPA. First-year 
cumulative GPA was a stronger predictor than family income. For every $10,000 increase in total family 
income, a student's odds of graduating increased by 1.06. Assuming a gap of $70,000 between the 
lowest and the highest family income, a student in the highest family income level would have a 50 
percent ^ higher chance of graduating than a student from the lowest family income level. Overall the 
final model that included both first-year cumulative GPA and family income increased the percent 
predicted correctly by 8 percent compared to the null model (i.e., a model with no variables) and had a 
Nagelkerke R^ of .306. 

* This is calculated by (1.06)''7 = 1.50, where 1.06 is the odds ratio and 7 equates to 70 thousand. 



University-wide, 22 percent of students had a NCOA greater than $20,000 (Table 3); however, 
the majority of these students were in the highest family income quintile. The mean (average) NCOA for 
all students was about $12,800 while the median (midpoint) was $12,600. Flagship Campus students 
had a higher mean NCOA ($15,200, see Table 4) than Residential Regional Campus students ($10,900, 
see Table 5) and Nonresidential Regional Campus students ($9,400, see Table 6). At the Flagship, 37 
percent of the students had a NCOA greater than $20,000, whereas only 12 percent of Residential 
Regional Campus students and 2 percent of Nonresidential Regional Campus students had a NCOA 
greater than $20,000. 

At the Flagship Campus (Table 7), with the exception of the students in the lowest family income 
quintile, for every increase of $10,000 in the NCOA, a student's chance of graduating decreased (odds 
ratio for the other income quintiles ranged from .72 to .96). The findings for the Residential Regional 
Campuses were relatively inconsistent. For students in the lowest, middle, and second-highest quintile, 
when the NCOA increased, the odds of graduating decreased; whereas, for the students in the second- 
lowest and highest-quintile, the odds of graduating increased as NCOA increased. These inconsistent 
results may be explained by the model's lack of a control variable for whether a student resided on 
campus or not. Except for students in the highest family income quintile (where NCOA had no effect on 
whether a student graduated or not), students at Nonresidential Regional Campuses had a lower chance 
of graduating as their NCOA increased. The effects did appear to vary among the family income levels as 
the students in the middle family quintile had the greatest effect (odds ratio of .62), while students in 
the second-lowest quintiles had the smallest effect (odds ratio of .90). Within all models, regardless of 
where a student starts at the University, first-year cumulative GPA was the strongest predictor of 
whether a student graduated or not. Overall, the addition of these two variables into the model 


increased the percent predicted correctly compared to the null model. 
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To examine whether any moderating effects existed between cumulative GPA and family 
income level, family income level was regressed on cumulative GPA (Table 10), controlling for total SAT. 
The models suggested that family income had little effect per $10,000 on a student's cumulative GPA lb- 
weight = .01) regardless of where a student started at the University. If we assume a $70,000 difference 
between the lowest family income quintile and the highest family quintile, a student in the highest 
family income quintile would have a .07 higher cumulative GPA than a student in the lowest family 
income quintile. The practical insignificance of the relationship between cumulative GPA and family 
income level suggests that the two variables might be mutually exclusive, which itself was an important 
finding. This finding indicates that even though family income and cumulative GPA were significant 
predictors of graduation, the effects of each variable were relatively independent of the other. 

We also examined the relationship between cumulative GPA and NCOA. At the Flagship Campus, 
the NCOA had a negligible effect on a student's cumulative GPA (Table 11) as the b-weights ranged from 
-.01 for students who did not complete a FAFSA to .07 for students in the lowest quintile. The positive b- 
weight for the lowest family income quintile might suggest that support and programs to help these 
students academically at the Flagship Campus are effective. The effects were more substantial for 
students at the Regional Campuses regardless of whether the campuses had housing or not (Table 12 
and Table 13). The b-weights ranged from -.25 to .11. A negative b-weight meant that as NCOA 
increased a student's cumulative GPA decreased. A concern is that the larger negative effects on 
cumulative GPA involved students in the second-lowest and middle quintile for Residential Regional 
Campus students. The negative effects were relatively large for all groups of Nonresidential Regional 
Campus students except for those in the highest family quintile. Utilizing the largest negative b-weight (- 
.25 for the middle family income quintile for Residential Regional Campus students) and assuming a 
NCOA of $20,000, this would correspond to a decrease in cumulative GPA of .5. Even though NCOA had 
practical significance, the variable explained 2.5 percent or less of the variance in cumulative GPA. 
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Lastly, we developed a regression model to examine the relationship between the time spent 
enrolled at the University and cumulative GPA and NCOA (Table 14) for students who did not graduate 
within six years (n = 3,639). Cumulative GPA had a positive effect on time spent enrolled (b-weight = .24) 
in that the higher the cumulative GPA, the more time a student who would not graduate within six years 
spent enrolled at the University. The NCOA, conversely, had a negative effect on time spent enrolled (b- 
weight = -.13), so if the student had a larger NCOA, she/he spent less time enrolled at the University. 
Conclusions 

Utilizing data on first-time, full-time, baccalaureate-seeking students who started in the summer 
or fall 2004, the current analyses find that academic performance (first-year cumulative GPA), family 
income, and NCOA are important predictors of graduation, validating the proposed model. An important 
finding is that the effects of cumulative GPA and family income on graduation are relatively independent 
of each other. Based on these findings, we posit that students from families with higher incomes are 
more likely to graduate than students from poor families, at least in part because they can afford the 
NCOA and persist to complete their degree requirements even in the face of poor academic 
performance. In examining the influence of NCOA on graduation, we find that it has differing effects at 
different family income levels. The findings mostly suggest that for students with less financial means, 
when NCOA increases, the influence on graduation and cumulative GPA is more negative than it is for 
students with more financial means. We also find that for non-graduates in the 2004 cohort, the higher 
the NCOA, the less time they spend enrolled. 

Understanding that paying tuition is necessary for a student to graduate provides a more 
plausible explanation for the graduation gaps between the varying family income levels and leads to the 
following hypothesis of success: The longer you are able to play the game, the more likely you are to 
succeed. This hypothesis is better illustrated with Mid-Atlantic Research I University students who have 


gone to non-degree status because of poor academic performance. Students who are dropped to this 
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status can continue attending the University for up to 30 credits as long as they earn at least a 2.01 GPA 
in any semester attended in non-degree status. This period is an opportunity for students to pull their 
cumulative GPA up to a level that allows them to be academically eligible to enter a major. 
Unfortunately, students in this status are ineligible for financial aid. Thus, students from families with 
high income who have entered this status can remain at the University because they have the resources 
to continue to pay the tuition, thereby having an opportunity to become eligible academically. Without 
financial aid, students from poor families may depart because they cannot afford the tuition to 
continue. Based on this logic, we hypothesize that the primary reason for the graduation gap that exists 
between the varying family income levels is simple: the wealthy have more financial resources than the 
poor, allowing them to remain enrolled until they graduate. Money then essentially buys time. 

Maintaining a certain cumulative GPA may also have financial implications for some students. 
Even though our models suggest that academic performance and family income are relatively 
independent of each other, we do think a real-world relationship exists between the two. Anecdotally, 
many students cite financial concerns as their reason for leaving the University without completing a 
degree. We hypothesize that in some cases the cause of their financial distress may be their failure to 
maintain a certain cumulative GPA, which in turn prevents them from qualifying for certain forms of aid. 

By having such a model, institutional researchers and administrators may be better able to 
develop and evaluate measures, policies, and programs that can improve the graduation rates at an 
institution. This model purposefully focuses on graduation; however, the difficulty in implementing 
strategies towards graduation is the time needed to assess the effectiveness and efficiencies of such 
plans (e.g., we need six years to see if graduation rates improve from cohort to cohort). Understanding 
the importance of academic performance and degree completion allows institutional researchers to 
develop Moneyball (Lewis, 2003) measures that are timelier than waiting six years to examine whether a 
student completed a degree or not. One such measure is monitoring the percentage and number of 
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students who earn a cumulative grade-point-average below 2.0 in their first year. The 2.0 cut-off is 
important at the University, because students cannot enter a major or graduate with a cumulative 
grade-point-average below it. Other hardships could also be incurred, such as a student entering into 
non-degree status, which prevents them from receiving federal student aid. This, then, becomes a more 
useful and timelier indicator to measure the effectiveness of programs that are geared towards 
improving graduation rates. 
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Table 1: 2004 Cohort Six-Year Graduation Rate by Family Income and 2004-2005 Cumulative GPA: University Wide 


Income 

Total Students 

2004-2005 Cumulative GPA 

No GPA 

<1.00 

1.00-1.99 

2.00-2.49 

2.50-2.99 

3.00-3.49 

3.50-4.00 

n 

6-year 
Grad Rate 

n 

6-year 

Grad 

Rate 

n 

6-year 

Grad 

Rate 

n 

6-year 

Grad 

Rate 

n 

6-year 

Grad 

Rate 

n 

6-year 

Grad 

Rate 

n 

6-year 

Grad 

Rate 

n 

6-year 

Grad 

Rate 

Did not file FAFSA Form 

2025 

74.3% 

20 

0.0% 

56 

7.1% 

126 

22.2% 

198 

53.5% 

442 

76.7% 

661 

85.0% 

522 

89.3% 

<$18,486 

640 

50.0% 

12 

8.3% 

46 

0.0% 

88 

15.9% 

95 

37.9% 

143 

57.3% 

148 

71.6% 

108 

75.0% 

$18,487-$34675 

1010 

53.8% 

15 

0.0% 

56 

3.6% 

134 

15.7% 

135 

40.7% 

243 

62.6% 

258 

68.6% 

169 

80.5% 

$34,676-$55230 

1613 

62.7% 

20 

0.0% 

53 

0.0% 

168 

21.4% 

253 

47.4% 

363 

69.4% 

441 

78.5% 

315 

81.9% 

$55,231-$88,002 

2557 

66.4% 

24 

0.0% 

62 

0.0% 

257 

24.5% 

328 

51.5% 

597 

68.0% 

731 

79.2% 

558 

86.0% 

>$88,002 

4367 

77.6% 

16 

0.0% 

65 

3.1% 

298 

32.2% 

445 

58.9% 

1002 

74.6% 

1405 

88.1% 

1136 

91.9% 

Total Students 

12212 

69.3% 

107 

0.9% 

338 

2.4% 

1071 

24.1% 

1454 

51.4% 

2790 

70.9% 

3644 

82.5% 

2808 

87.8% 


Table 2: Logistic Regression Model - Predicting Six-year Graduation for All Students (n=12,105) 



Model 1 

Model 2 

Model 3 

Model Variables’ 




Constant 

.030 

3.01 

.04 

First-year cumulative GPA 

4.69 

- 

4.49 

Completed FAFSA 

- 

.38 

.49 

Completed FAFSA and Family Income (10k) 

- 

1.08 

1.06 





Model 




Correct Predicted 

77.5% 

69.9% 

77.5% 

Nagelkerke R^ 

.289 

.048 

.306 


Null model correct predicated = 69.9% 
a: Odds ratio presented 
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Table 3: Distribution of First-year Net Cost of Attendance by Family Income (University-Wide) 



Net Cost of Attendance | 


n 

Less than 
$0 

$0k-$5k 

$5k -$10k 

$10k-$15k 

$15k-$20k 

More than 
$20k 

Mean 
(in $10k) 

Median 
(in $10k) 

Did Not file 
FAFSA 

2025 

.9% 

1.7% 

2.6% 

7.5% 

25.2% 

62.0% 

2.07 

2.07 

<$18,486 

640 

3.8% 

37.5% 

26.7% 

17.2% 

8.3% 

6.6% 

0.79 

0.61 

$18,487- 

$34,675 

1010 

3.6% 

38.8% 

32.5% 

14.8% 

6.7% 

3.7% 

0.70 

0.58 

$34,676- 

$55,230 

1613 

2.0% 

33.2% 

31.3% 

18.7% 

10.4% 

4.3% 

0.83 

0.71 

$55,231- 

$88,002 

2557 

1.3% 

27.5% 

23.5% 

23.4% 

15.9% 

8.3% 

1.01 

0.94 

>$88,002 

4367 

1.0% 

19.2% 

12.8% 

14.8% 

26.9% 

25.3% 

1.43 

1.56 

Total 

12212 

1.5% 

22.5% 

18.2% 

16.0% 

19.5% 

22.3% 

1.28 

1.26 


Table 4: Distribution of First-year Net Cost of Attendance by Family Income (Flagship Campus) 



Net Cost of Attendance | 


n 

Less than 
$0 

$0k-$5k 

$5k -$10k 

$10k-$15k 

$15k-$20k 

More than 
$20k 

Mean 
(in $10k) 

Median 
(in $10k) 

Did Not file 
FAFSA 

1287 

1.5% 

1.8% 

2.4% 

6.8% 

8.4% 

79.1% 

2.25 

2.09 

<$18,486 

203 

5.4% 

40.9% 

18.2% 

14.3% 

8.9% 

12.3% 

0.85 

0.59 

$18,487- 

$34,675 

298 

7.0% 

39.9% 

24.8% 

13.1% 

7.0% 

8.1% 

0.73 

0.54 

$34,676- 

$55,230 

552 

4.3% 

38.2% 

22.5% 

15.8% 

10.5% 

8.7% 

0.84 

0.65 

$55,231- 

$88,002 

1006 

1.4% 

33.8% 

15.5% 

16.2% 

17.3% 

15.8% 

1.09 

0.98 

>$88,002 

2566 

1.4% 

19.1% 

8.4% 

8.8% 

26.4% 

35.9% 

1.61 

1.82 

Total 

5912 

2.1% 

21.4% 

10.8% 

10.7% 

17.9% 

37.1% 

1.52 

1.72 


Table 5: Distribution of First-year Net Cost of Attendance by Family Income (Residential Regional Campuses) 



Net Cost of Attendance | 


n 

Less than 
$0 

$0k-$5k 

$5k -$10k 

$10k-$15k 

$15k-$20k 

More than 
$20k 

Mean 
(in $10k) 

Median 
(in $10k) 

Did Not file 
FAFSA 

459 

0.0% 

1.1% 

2.8% 

5.7% 

42.5% 

47.9% 

1.88 

1.99 

<$18,486 

256 

2.3% 

35.5% 

28.5% 

19.9% 

8.6% 

5.1% 

0.80 

0.67 

$18,487- 

$34,675 

404 

2.7% 

35.4% 

33.2% 

16.3% 

9.7% 

2.7% 

0.75 

0.65 

$34,676- 

$55,230 

665 

.9% 

32.6% 

32.5% 

19.7% 

11.7% 

2.6% 

0.83 

0.74 

$55,231- 

$88,002 

988 

1.5% 

27.3% 

24.0% 

24.1% 

18.5% 

4.6% 

0.97 

0.94 

>$88,002 

1235 

.5% 

23.4% 

14.7% 

17.2% 

30.3% 

14.0% 

1.22 

1.34 

Total 

4007 

1.1% 

25.3% 

21.3% 

18.1% 

22.2% 

12.0% 

1.09 

1.04 
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Table 6: Distribution of First-year Net Cost of Attendance by Family Income (Nonresidential Regional Campuses) 



Net Cost of Attendance | 


n 

Less than 
$0 

$0k-$5k 

$5k-$10k 

$10k-$15k 

$15k-$20k 

More than 
$20k 

Mean 
(In $10k) 

Median 
(in $10k) 

Did Not file 
FAFSA 

259 

0.0% 

2.3% 

3.5% 

14.7% 

75.7% 

3.9% 

1.52 

1.56 

<$18,486 

169 

4.1% 

39.1% 

36.1% 

13.0% 

5.3% 

2.4% 

0.66 

0.54 

$18,487- 

$34,675 

293 

1.4% 

44.4% 

39.6% 

11.9% 

2.4% 

.3% 

0.59 

0.55 

$34,676- 

$55,230 

376 

.8% 

28.7% 

43.6% 

19.7% 

5.9% 

1.3% 

0.77 

0.69 

$55,231- 

$88,002 

539 

.7% 

17.3% 

38.2% 

35.3% 

7.6% 

.9% 

0.90 

0.86 

>$88,002 

550 

.2% 

10.7% 

29.3% 

37.1% 

21.5% 

1.3% 

1.08 

1.20 

Total 

2186 

.9% 

21.1% 

32.8% 

25.8% 

18.0% 

1.5% 

0.94 

0.89 


Table 7: Six-year Graduation Logistic Regression Model for Flagship Campus by Family Income 



Did Not fill 
FAFSA 

<$18,486 

$18,487- 

$34,675 

$34,676- 

$55,230 

$55,231- 

$88,002 

>$88,002 

Model Variables^ 







Constant 

.08 

.02 

.13 

.23 

.05 

.04 

First-year cumulative GPA 

4.38 

5.59 

3.23 

4.33 

5.41 

5.75 

NCOA (per 10k) 

.97 

1.06 

.72 

.78 

.75 

.96 








Model Evaluation 







Correct Predicted 

88.2% 

80.8% 

78.5% 

84.2% 

87.2% 

89.6% 

Nagelkerke R'^ 

.184 

.361 

.167 

.225 

.259 

.221 

Null Model Correct 
Predicted 

86.6% 

70.9% 

74.5% 

80.6% 

83.2% 

88.2% 

n 

1287 

203 

298 

552 

1006 

2566 


a: Odds ratio presented 


Table 8: Six-year Graduation Logistic Model for Residential Regional Campuses by Family Income 



Did Not fill 
FAFSA 

<$18,486 

$18,487- 

$34,675 

$34,676- 

$55,230 

$55,231- 

$88,002 

>$88,002 

Model Variables’ 







Constant 

.04 

.04 

.01 

.04 

.09 

.04 

First-year cumulative GPA 

3.48 

3.24 

4.72 

4.08 

3.13 

3.70 

NCOA (per 10k) 

1.24 

.80 

1.20 

.64 

.80 

1.38 








Model Evaluation 







Correct Predicted 

73.4% 

63.3% 

70.8% 

71.9% 

69.6% 

73.4% 

Nagelkerke 

.228 

.233 

.329 

.293 

.201 

.227 

Null Model Correct Predicted 

62.1% 

56.6% 

50.0% 

57.4% 

61.6% 

67.7% 

n 

459 

256 

404 

665 

988 

1235 


a: Odds ratio presented 
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Table 9: Six-year Graduation Logistic Model for Nonresidential Regional Campuses by Family Income 



Did Not fill 
FAFSA 

<$18,486 

$18,487- 

$34,675 

$34,676- 

$55,230 

$55,231- 

$88,002 

>$88,002 

Model Variables’ 







Constant 

<.01 

.01 

.03 

.04 

.01 

.03 

First-year cumulative GPA 

3.42 

4.69 

3.36 

3.62 

4.79 

3.65 

NCOA (per 10k) 

6.54 

.73 

.90 

.62 

.85 

1.00 








Model Evaluation 







Correct Predicted 

67.2% 

69.2% 

65.9% 

71.0% 

70.1% 

68.4% 

Nagelkerke R'^ 

.288 

.363 

.260 

.249 

.296 

.233 

Null Model Correct Predicted 

59.5% 

62.1% 

59.4% 

50.8% 

53.4% 

52.9% 

n 

259 

169 

293 

376 

539 

550 


a: Odds ratio presented 


Table 10: First-year CGPA OLS Regression Models with SAT and Family Income as Predictors 



University- 

Wide 

Flagship 

Campus 

Residential Regional 
Campuses 

Nonresidential 
Regional Campuses 

Constant 

2.05 

2.44 

2.17 

1.95 

Total SAT score (per 100) 

.09 

.06 

.06 

.06 

Completed FAFSA 

-.17 

-.16 

-.07 

.07 

Completed FAFSA and Family Income (per 10k) 

.01 

.01 

.01 

.01 






Adjusted R'^ 

.077 

.054 

.021 

.024 


Table 11: First-year CGPA OLS Regression Models by Family Income Levels (Flagship Campus) 



Did Not fill 
FAFSA 

<$18,486 

$18,487- 

$34,675 

$34,676- 

$55,230 

$55,231- 

$88,002 

>$88,002 

Constant 

3.18 

2.80 

2.95 

2.98 

3.04 

3.16 

NCOA (per 10k) 

-.01 

.07 

.01 

.01 

.02 

.01 








Adjusted R'^ 

.000 

.000 

.000 

.000 

.000 

.000 


Table 12: First-year CGPA OLS Regression Models by Family Income Levels (Residential Regional Campuses) 



Did Not fill 
FAFSA 

<$18,486 

$18,487- 

$34,675 

$34,676- 

$55,230 

$55,231- 

$88,002 

>$88,002 

Constant 

2.56 

2.61 

2.78 

2.92 

2.77 

2.80 

NCOA (per 10k) 

.11 

-.01 

-.21 

-.25 

.03 

.02 








Adjusted R^ 

.002 

.000 

.014 

.025 

.000 

.000 


Table 13: First-year CGPA OLS Regression Models by Family Income Levels (Nonresidential Regional Campuses) 



Did Not fill 
FAFSA 

<$18,486 

$18,487- 

$34,675 

$34,676- 

$55,230 

$55,231- 

$88,002 

>$88,002 

Constant 

2.88 

2.54 

2.71 

2.79 

2.85 

2.66 

NCOA (per 10k) 

-.24 

-.22 

-.33 

-.12 

-.20 

.03 








Adjusted R'^ 

.001 

.005 

.013 

.001 

.009 

.000 


Table 14: OLS Regression Models for Length of time for Students who did not Graduate within Six Years 


(n=3,639) 



Model 1 

Model 2 

Constant 

2.36 

2.50 

First-year cumulative GPA 

.24 

.24 

NCOA (10k) 

- 

-.13 




Adjusted R^ 

.017 

.019 









1 

THE RIPPLE EFFECT: DECLINING STUDENT INTEREST IN HUMANITIES 
AMIDST GROWING DEMAND FOR PROFESSIONAL EDUCATION 

Elissa C. Lu, Ph.D. 

Analyst, Office of Institutional Research, Office of the President and Provost 

Harvard University 
Working brief 

Abstract 

As students and policymakers increasingly view college as a steppingstone to the 
professional labor market, scholars and pundits have noted flagging student interest in the 
humanities. This working brief examines the degree to which humanities degree conferrals are 
dropping nationally and describes an initiative one institution undertook to curtail declining 
undergraduate enrollments. A review of publicly available data found that degree conferrals in 
the humanities experienced the steepest decline in the 1960s and that changes can be partially 
attributed to demographic shifts. Compared to other institutions. Ivy League and Top 50 ranked 
US News and World Report institutions have experienced stronger shifts away from the 
humanities and greater movement toward natural sciences and other fields over the past ten 
years. The findings also reveal variation in the proportion of degrees awarded by major within 
humanities and uneven student interest across pre-professional programs. To address decreasing 
student enrollments and concentrations in its humanities division. Harvard’s Humanities Project 
outlined a multi-faceted approach for the Division to more effectively reach and interest students 
in humanities coursework and engage students in new, innovative forms of learning. Harvard’s 
experience suggests that shifting enrollments creates opportunities to revisit teaching and the role 
of real-world skill development. The case study also demonstrates how institutional research 
offices can offer leadership by helping administrators understand core issues and data behind 
changes in enrollments, and by tracking enrollments and students’ learning experiences by 
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division. Moving forward, institutional research offices have an opportunity to assess student and 
graduate outcomes across divisions and contribute to the national conversation on the 
relationship between humanities and career preparation. 
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Soaring tuition, rising student debt, and a post-recessionary economy have revived 
debates about the role of humanities and job preparation in higher education. Students are 
increasingly mindful of the connection between the labor market and their postsecondary 
education: the annual Higher Education Research Institute (HERI) survey found that an all-time 
high proportion of incoming freshmen think it is very important to go to college to “make more 
money” (74.6%) and to get a better job (87.9%; Pryor, Eagen, Palucki Blake, Hurtado, Berdan, 
& Case, 2012). However, approximately half of new graduates are unemployed or 
underemployed (Vedder, Denhart, & Robe, 2013). 

In the current economy and policy climate, humanities and liberal arts have seemingly 
fallen out of favor with the public. The media and policy and think tank reports often feature 
humanities and liberal arts as “worst” majors given their employment and salary prospects 
(Camevale, Cheah, & Strohl, 2012; Camevale, Strohl, & Melton, 2011; Goudreau, 2012; 
Rapacon, 2013). Earlier this year, Elorida’s Blue Ribbon Task Eorce on State Education Reform 
recommended that students enrolled in low-demand majors pay higher tuition than their peers. It 
is plausible that more students are gravitating away from the humanities and liberal arts and 
toward other, more lucrative fields and career paths (Wilson, 2012). 

News headlines articulate colleges’ and universities’ concerns over the shift away from 
humanities (e.g., October 2013 New York Times article “As interest fade in the humanities, 
colleges worry”). In June 2013, 160 employers and 107 college presidents signed an Association 
of American Colleges and Universities (AACU) compact promoting the significance of a 2U* 
century liberal arts education to the public and on their campuses. Proponents of a liberal arts 
education posit that higher education should not be viewed in solely instrumental or economic 
terms. A liberal arts education offers students the ability to develop their critical thinking skills. 
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creativity, and awareness of their environments - skills and abilities that will guide students 
throughout the course of their lives and promote the advancement of our society. These 
generalizable skills also assist students in the labor market and in an economy that is 
experiencing rapid technological change and globalization (McPherson, 1998). 

Yet amidst students’ and policymakers’ concerns about the economic value of a college 
education, it is important for higher education leaders to understand how student demand for the 
humanities has recently shifted, if at all. Additionally, it is helpful to garner a sense of 
institutional responses to changing student interests: what are specific ways institutions have 
contended with new student demands? This working brief synthesizes national data and 
published reports and articles to offer a descriptive view of fluctuating student interest in the 
humanities and serve as a starting point for discussion and future research. 

Dispelling the Humanities “Crisis” 

Headlines often denote waning interest in humanities (Brooks, 2013; Siegel, 2013). Many 
articles cite the precipitous drop in humanities majors from 1966 - when one in six (17%) 
degrees were awarded in the humanities - to 2011, when one in twenty (6%) degrees were in the 
humanities according to American Academy of Arts and Sciences (AAAS) definitions. However, 
as scholars note, this seemingly precipitous drop in humanities degree conferrals may be 
misleading and be more reflective of shifts in demography and the growth of other fields. How 
one frames analysis and investigates the underlying assumptions shape our understanding of 
trends in humanities degrees. 

The baseline year, for instance, impacts how much degrees appear to decline. Ben 
Schmidt (2013), a former fellow at the AAAS, notes that recordkeeping has contributed to the 
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perception that student interest in humanities has dipped. NCES had not digitized earlier degree 
conferral data, leading many to anchor reporting about the humanities enrollments in the mid- 
1960s, which is coincidentally the highest point of degree conferrals in the humanities. A review 
of historical data assembled by AAAS from the 1940s onward reveals that enrollments have 
seesawed, and that the decline was sharpest from the late 1960s and 1970s to the mid-1980s. The 
proportion of bachelor’s degrees awarded in the humanities has remained relatively stable over 
the past twenty years. 


Figure 1. Bachelor’s degrees awarded in the humanities 





■ Core 

■ CIP 


Source: American Academy of Arts and Sciences. (2013). Supporting Table Il-la: Bachelor’s Degree Completions 
in the Humanities (Absolute Number and as a Percentage of All Bachelor’s Degree Completions), 1948-2011. 
Humanities Indicators, humanitiesindicators.org. AAAS uses two methodologies (Core and CIP) to aggregate 
humanities majors over time. 

Given the surge of older working students who are more likely to seek out career-oriented 
programs, statistician Nate Silver (2013) argues that the percentage of bachelor’s degrees 
awarded in the humanities has actually remained relatively stable over time when only 
considering the traditional- aged student population. According to his calculations, there has been 
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a less than a one percent change from 1971 to 2011 in the distribution of bachelor’s degrees 
awarded in English as well as other subject areas such as statistics, engineering, and social 
sciences and history among students aged 24 and younger. Health professions, business, and 
criminal justice are majors that have experienced the greatest growth. Hence, the declining 
proportion of degree conferrals in the humanities can be attributed to a growing number of adults 
enrolling in higher education who are more likely to be working and looking for career 
advancement. Looking ahead, the proportion of adult students will only continue to grow: NCES 
(2012) estimates that student enrollments under the age of 25 will increase 10% compared to 
20% for students age 25 and older in 2020 (see Eigure 2). 


Figure 2. Total fall enrollment in degree-granting institutions by student age and 
proportion of enrollments 25 and older, select years 


25.000 

20.000 

15.000 

10.000 
5,000 



Other scholars like Ben Schmidt have noted that the dip in humanities degree conferrals 
also corresponds with women’s increasing participation in higher education and exodus from the 
humanities to other fields. Eor instance, according to NCES data, in 1970, 43% of bachelor’s 
degrees were awarded to women; at that time, 16% of women majored in English and foreign 
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languages. By 1980, nearly half (49%) of degrees were awarded to women and the percentage 
majoring in English and foreign languages dropped dramatically (7%). 

One also needs to consider variation within the humanities: individual majors have 
differed in their rate of growth and decline over time. English, for example, constituted 45% of 
bachelor’s degrees awarded in the humanities in 1970-1971, but only 18% in 2010-2011. In 
contrast, visual and performing arts increased from 21% to 33% of all humanities degrees 
awarded during that same time period. 

Have Degrees Shifted by Fields in The Past Ten Years? 

Erom an institutional decision-making perspective, it is helpful to hone in on trends in the 
past ten years. As such, descriptive statistics were generated for 3,083 U.S. institutions that 
conferred bachelor’s degrees as reported to the National Center for Education Data Set (NCES) 
Integrated Postsecondary Education Data Set (IPEDS) in 2002-2003 and 201 1 -20 1 2 to explore 
trends over a ten-year timeframe. Approximately half of the institutions are private, not-for-profit 
(51.9%) and the remainder is for-profit (25.6%) and public, not-for-profit (22.4%). A small 
proportion (5.1%) of institutions has a sole Arts and Science focus according to the Carnegie 
2010 classification of undergraduate instruction. Analysis examined shifts in distribution across 
the 2003 to 2012 academic years by field and two-digit Classification of Instructional Program 
codes (CIP), using NCES definitions. 

Although scholars have noted that humanities degree conferrals have remained relatively 
stable in recent years, it is important to note that there has been substantial growth during the 
past ten years in bachelor’s degrees conferred overall and across all disciplines. Among the 
institutions of focus, bachelor’s degree conferrals grew 33% from 1,343,284 in 2002-2003 to 



1,793,157 in 2011-2012 and at a compound annual growth rate of 3.26%. By sector, for-profit 
colleges and universities exhibited the strongest growth, tripling their degree conferrals during 
this time period (307%) while public institutions grew 29.7% and private, non-profit grew 
20.5%. As Table 1 highlights, the number of degrees awarded grew across nearly all fields; the 
rate of growth was highest in the “other fields” category (6.1% CAGR) followed by natural 
sciences (5.1%), and social and behavioral sciences (3.0%) (based on NCES classification of 
two-digit CIPs into disciplines). The number of humanities degrees awarded increased 24.5% 
over the time period, or at a 2.5% annual growth rate. 


Table 1. Number and distribution of bachelor’s degrees conferred by field, 2002-2003 and 

2011-2012 


Field 

2003 

Degrees 

2012 

Degrees 

2003 

(%) 

2012 

(%) 

Change 

from 

2003-2012 

(%) 

CAGR 

Business 

290,899 

367,414 

21.7% 

20.5% 

-1.2% 

2.6% 

Computer sciences and engineering 

134,205 

146,045 

10.0% 

8.1% 

-1.8% 

0.9% 

Education 

104,791 

105,846 

7.8% 

5.9% 

-1.9% 

0.1% 

Humanities 

237,423 

295,723 

17.7% 

16.5% 

-1.2% 

2.5% 

Natural Sciences 

90,332 

141,488 

6.7% 

7.9% 

1.2% 

5.1% 

Other fields 

264,371 

448,958 

19.7% 

25.0% 

5.4% 

6.1% 

Social and behavioral sciences 

221,263 

287,683 

16.5% 

16.0% 

-0.4% 

3.0% 

Total 

1,343,284 

1,793,157 

100% 

100% 


3.3% 


As Table 1 also indicates, the distribution of degrees awarded over the past ten years has 
moved toward fields “other” than business, computer sciences, education, and the humanities. 
Degrees in health-related professions underlie much of the growth in the “other fields” category; 
they represented 5.2% of bachelor’s degrees in 2003 and 9.1% of all bachelor’s degrees in 2012 
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(see Table 2). Within the other fields category, the share of degrees awarded in homeland 
security also increased from 26,022 degrees (or 1.9% of all bachelor’s degrees) in 2003 to 53,807 
(or 3.0% of all bachelor’s degrees) in 2012. Among other fields and majors, biological and 
biomedical sciences increased from 59,928 degrees (4.5%) to 95,935 (5.4% of bachelor’s 
degrees). 

On the whole, humanities degrees declined slightly (-1.2%) as a proportion of overall 
bachelor’s degrees. Most humanities majors (i.e., English, liberal arts, philosophy, theology) 
either decreased a little or experienced no change (i.e., foreign languages and area, ethnic, 
cultural, gender, and group studies). Only two humanities majors increased their share of overall 
bachelor’s degrees: multi-interdisciplinary studies (.4% increase in share; 5.5% CAGR) and 
visual and performing arts (.1% increase in share, 3.4% CAGR). 

Similar to humanities, three other professionally- aligned fields, education and computer 
sciences and engineering, and business, also experienced decreases in the share of overall 
degrees awarded. Over the past ten years, the share of education degrees declined from 7.8% of 
bachelor’s degrees in 2003 to 5.9% of bachelor’s degrees in 2012. Additionally, computer and 
information sciences dropped slightly from 56,940 (4.2% of all bachelor’s degrees) to 47,422 
(2.6%) in 2012. The number of bachelor’s degrees awarded in business, management, marketing, 
and related areas also represented a smaller proportion of all degrees (20.5%) in 2012 than in 


2003 (21.7%). 
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Table 2. Bachelor’s degrees conferred by field, 2002-2003 and 2011-2012 


Field 

Major 

2003 

2012 

% of 

% of 

% change in 

CAGR 





2003 


distribution 






total 

total 

2003-2012 


Business 

Business, Management, Marketing, and Related 

290,371 

366,079 

21.6% 

20.4% 

-1.2% 

2.7% 


Support Services 

Personal and Culinary Services 

528 

1,335 

0.0% 

10.0% 

0.0% 

10.9% 

Computer 

Engineering 

62,713 

81,460 

4.7% 

4.5% 

-0.1% 

3.2% 

sciences and 

Computer and Information Sciences and Support 

56,940 

47,422 

4.2% 

2.6% 

-1.6% 

-0.5% 

engineering 

Services 







Engineering Technologies and Engineering- 
related Fields 

14,233 

16,536 

1.1% 

0.9% 

-0.1% 

2.3% 


Construction Trades 

166 

377 

0.0% 

0.0% 

0.0% 

9.5% 


Mechanic and Repair Technologies/Technicians 

153 

250 

0.0% 

0.0% 

0.0% 

5.6% 

Education 

Education 

104,791 

105,846 

7.8% 

5.9% 

-1.9% 

0.9% 

Humanities 

Visual and Performing Arts 

71,026 

95,834 

5.3% 

5.3% 

0.1% 

3.4% 


English Language and Literature 

53,606 

53,819 

4.0% 

3.0% 

-1.0% 

0.6% 


Liberal Arts and Sciences General Studies and 
Humanities 

42,323 

46,996 

3.2% 

2.6% 

-0.5% 

1.5% 


Multi-Interdisciplinary Studies 

28,391 

45,803 

2.1% 

2.6% 

0.4% 

5.5% 


Foreign Languages Literatures and Linguistics 

16,923 

21,783 

1.3% 

1.2% 

0.0% 

2.9% 


Philosophy and Religious Studies 

10,342 

12,746 

0.8% 

0.7% 

-0.1% 

2.4% 


Theology and Religious Vocations 

8,206 

9,510 

0.6% 

0.5% 

-0.1% 

1.7% 


Area Ethnic, Cultural, Gender and Group Studies 

6,606 

9,232 

0.5% 

0.5% 

0.0% 

3.8% 

Natural Sciences 

Biological and Biomedical Sciences 

59,928 

95,935 

4.5% 

5.4% 

0.9% 

5.4% 


Physical Sciences 

17,727 

26,118 

1.3% 

1.5% 

0.1% 

4.4% 


Mathematics and Statistics 

12,497 

18,875 

0.9% 

1.1% 

0.1% 

4.7% 


Science Technologies/Technicians 

180 

560 

0.0% 

0.0% 

0.0% 

13.4% 

Other fields 

Health Professions and Related Programs 

70,424 

163,837 

5.2% 

9.1% 

3.9% 

9.8% 


Communication Journalism and Related 

67,776 

83,819 

5.0% 

4.7% 

-0.4% 

2.5% 


Programs 

Homeland Security, Law Enforcement, 
Firefighting and Related Professions 

26,022 

53,807 

1.9% 

3.0% 

1.1% 

8.4% 


Parks Recreation Leisure and Fitness Studies 

21,352 

38,998 

1.6% 

2.2% 

0.6% 

6.9% 


Public Administration and Social Service 
Professions 

19,783 

29,710 

1.5% 

1.7% 

0.2% 

4.6% 


Family and Consumer Sciences/Human Sciences 

17,704 

23,428 

1.3% 

1.3% 

0.0% 

3.3% 


Agriculture Operations and Related Sciences 

14,188 

16,252 

1.1% 

0.9% 

-0.1% 

1.7% 


Natural Resources and Conservation 

9,055 

14,687 

0.7% 

0.8% 

0.1% 

5.5% 


Architecture and Related Services 

8,983 

9,728 

0.7% 

0.5% 

-0.1% 

1.9% 


Communications Technologies/Technicians and 

1,933 

4,982 

0.1% 

0.3% 

0.1% 

11.1% 


Support Services 

Transportation and Materials Moving 

4,635 

4,896 

0.3% 

0.3% 

-0.1% 

1.2% 


Legal Professions and Studies 

2,369 

4,596 

0.2% 

0.3% 

0.1% 

7.6% 


Library Science 

99 

95 

0.0% 

0.0% 

0.0% 

0.1% 


Military Technologies and Applied Sciences 

6 

86 

0.0% 

0.0% 

0.0% 

34.4% 


Precision Production 

42 

37 

0.0% 

0.0% 

0.0% 

0.0% 

Social and 

Social Sciences 

115,297 

143,511 

8.6% 

8.0% 

-0.6% 

2.6% 

behavioral 

sciences 

Psychology 

78,238 

109,021 

5.8% 

6.1% 

0.3% 

3.8% 


History 

27,728 

35,151 

2.1% 

2.0% 

-0.1% 

2.8% 

Total 


1,343,284 

1,793,157 

100.0% 

100.0% 


3.3% 
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However, shifts in the distribution of degree conferrals varied across groups of 
institutions. As Figure 3 demonstrates, Ivy League and top 50 ranked US News and World 
Report national universities experienced stronger movement in their overall distribution away 
from the humanities and social and behavioral sciences and toward the natural sciences 
compared to all colleges and universities in the 2003 to 2012 academic years. 


Figure 3. Distribution of bachelor degree conferrals at Ivy League, Top 50 US News and 
World Report (USWNR) national universities and all other institutions by field, 2002-2003 

and 2011-2012 



■ All Institutions 

■ USNWR Top 50 National 
Universities 

■ Ivy League 


Note; Ivy League institutions include Brown University, Columbia University, Cornell University, Dartmouth 
University, Harvard University, Princeton University, University of Pennsylvania, and Yale University. Top 50 
USNWR universities are based on 2014 rankings. 

The Humanities Project: One Institution’s Response to Declining Enrollments 

Corresponding with the finding that certain groups of institutions have faced sharp 
declines in their humanities degree conferrals, one such institution. Harvard University, sought to 
draw the public’s attention to the importance of the humanities and to boost its own students’ 
interest in the field. This section describes how Harvard’s Humanities Project initiative was 
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formed and implemented and is based on a review of articles and reports and interviews with 
faculty and administration involved in the Project. 

Context. Founded in 1636, Harvard is one of the oldest universities in the United States. 
Humanities have had a long tradition at the University, with approximately 40% of the 2,275 
full-time equivalent faculty based at the Faculty of Arts and Sciences (FAS) and approximately 
42% of its $32.7 billion endowment belonging to FAS. Over one-third (35%) of degrees are 
awarded from FAS which includes Harvard College and the Graduate School of Arts and 
Sciences. While Harvard is a unique institution in several regards, the University serves as a 
bellwether in higher education given its position as one of the oldest, most prestigious 
universities in the nation. 

As a large, decentralized university with 14 degree granting schools. Harvard University 
had previously engaged in several university- wide initiatives to explore the role of arts and 
humanities on campus and to foster cross-school collaboration. Recent efforts have included the 
Arts Task Force, Harvard University Council of Arts, Report on the Study of Religion, and 
development of its American Repertory Theater. All of the initiatives have explored ways to 
make aspects of arts and humanities more relevant on campus, including to undergraduate 
students. 

President Drew Faust has also published op-eds and delivered speeches calling attention 
to the non-economic value of higher education amidst growing public interest in college 
graduates’ employability. In a speech delivered at the Royal Irish Academy in 2010, President 
Faust noted the importance of developing students’ interpretation skills and curiosity through a 


liberal arts education: 
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When we define higher education's role principally as driving economic development and 
solving society's most urgent problems, we risk losing sight of broader questions, of the 
kinds of inquiry that enable the critical stance, that build the humane perspective, that 
foster the restless skepticism and unbounded curiosity from which our profoundest 
understandings so often emerge... 

An overly instrumental model of the university misses the genius of its capacity. It 
devalues the zone of patience and contemplation the university creates in a world all but 
overwhelmed by stimulation. It diminishes its role as an asker of fundamental questions 
in a world hurrying to fix its most urgent problems. We need both. 

Similarly, in a February 2013 op-ed to the New York Times, President Faust argued that the merit 

of a postsecondary education was being overshadowed by employment concerns: “The focus in 

federal policy making and rhetoric on earnings data as the indicator of the value of higher 

education will further the growing perception that a college degree should be simply a ticket to a 

first job, rather than a passport to a lifetime of citizenship, opportunity, growth and change.” 


Arts and Humanities Division Initiative. Over the past several years, faculty and 
administration in the Arts and Humanities Division of FAS have observed fewer students in their 
courses, which are already designed to be small and enable intensive interactions between 
students and faculty (e.g., in 201 1-2012, 44% of classes at the College had nine or fewer 
students). Whereas approximately half of Harvard College students concentrated in the 
humanities in the 1920s, that percentage has drastically decreased over time. Meanwhile, the 
faculty has watched the University’s efforts to bolster sciences and engineering with great 
interest. In 2007, the Division of Sciences and Engineering officially became the School of 
Engineering and Applied Sciences (SEAS), the first school the University formed in decades. 
SEAS has grown steadily since: the number of faculty increased from 43 to approximately 70 


ETEs with plans to expand to over 100 ETE faculty and a new building on the Allston campus in 
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the coming years. Among students, the number of computer science, applied mathematics, and 
biomedical engineering concentrators has grown over 10% in the past two years (Zhang, 2013). 

After many conversations with department heads, in 201 1, the Dean of the Arts and 
Humanities Division decided that it was time to proactively address the role of humanities at the 
university and explore what actions the Division might take. The Dean called together two 
faculty working groups; one working group explored the role of humanities in the university 
mission and its history and the other working group examined how the curriculum could excite 
students’ interest and promote departmental collaboration. Each group, chaired by two faculty 
members, was to generate a set of recommendations by the end of the following academic year, 
or a year and a half later. Working groups set ambitious agendas for themselves and developed 
reading lists of major works and examined efforts at other institutions to inform their actions. 
Each group was staffed by a graduate student to help the committees complete their work within 
a year and a half. 

Role of Institutional Data. Institutional data played an important role in helping the 
working groups understand the magnitude of decline in enrollments and concentrators in the 
humanities division, as well as make the case to other faculty and the public about the need for 
intervention. Institutional data also played a critical role in guiding recommendations to the 
Division. Harvard College’s institutional research office conducted in-depth analyses about 
humanities concentrators relative to other divisions by exploring various survey data and 
institutional data. ^ Among questions the office explored: How have course enrollments and 
student concentrations changed by division over time? Do students on financial aid tend to 

^ There are two institutional research offices at Harvard. Harvard College’s institutional research office focuses 
primarily on the undergraduate population. The author is situated in the University institutional research office under 
the Office of the President and Provost. 
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concentrate in different departments? Do peer institutions face similar trends in degree conferrals 
by field? What are the motivations behind students’ major selection? How does student 
persistence in and satisfaction with concentration vary across divisions? 

Results illustrated the steep decline in the number of humanities concentrators from 36 to 
20 percent over the past 60 years (when including the History department). From 2001 to 201 1, 
enrollment in general education courses fell by 9% while enrollment in science courses climbed 
12%. Yet humanities concentrators rated high levels of satisfaction with their concentrations 
compared to students from other divisions. Importantly, data revealed that approximately half of 
students who entered the College with hopes of concentrating in the humanities ultimately ended 
up concentrating in another division, and most frequently the social sciences. The proportion of 
students leaving the Arts and Humanities Division surpassed other divisions. The committees 
realized that the first three semesters were a critical period for the Arts and Humanities Division 
to retain students before they declared their concentration. 

Humanities Project recommendations. The committees put forth numerous suggestions, 
some of which have already been implemented. One of the major changes has been the addition 
of new courses. Given the critical period between students’ entrance and declaration of their 
concentration, the Committee recommended that a set of introductory “framework” courses be 
developed. Although the Division had a plethora of courses that offered small class sizes and 
close interaction with faculty, members of the committee acknowledged that this class structure 
might not appeal immediately to all students. With strong instruction and interesting content, the 
courses were designed to be meaningful learning experiences and to encourage students to 
continue taking humanities courses or concentrating in the humanities. These courses, “The Art 
of Listening,” “The Art of Reading,” and “The Art of Looking” are interdisciplinary and seek to 
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incorporate student engagement. In The Art of Looking, for instance, students learn about 
different ways of thinking and important works through a range of technologies that span time, 
culture, and medium. Some of these technologies include television, film, glass, mahogany, 
color, and graphs. As the course syllabus notes: 


The Art of Looking aims to give students a solid grounding in the techniques of 
humanistic research and analysis, as well as an understanding of how humanistic thinking 
can transform civic participation, social experience, and everyday life. It will foster skills 
in analytic thinking, verbal and visual communication, and creative problem solving that 
will prepare students to be more effective students, citizens, and professionals across a 
broad spectrum of fields. 

Each week, students spend one session in lecture, section, and a hands-on learning laboratory. 


Division leaders also discussed the importance of changing students’ perceptions about 
the humanities. To confront beliefs that college concentrations should directly align with post- 
graduation employment, the Division sought to communicate that humanities concentrators 
could continue onto any profession they desired. One way to communicate this message was 
through a redeveloped website. Previously, the website was a basic, internal site (“iSite”) that 
served as a portal to individual departments. Under the direction of a staff member in the Dean’s 
Office, the new website was meant to be a “most beautiful museum” that invited visitors to 
explore information on the site. Select alumni were profiled to highlight how their humanities 
coursework helped prepare them for careers in a variety of fields, such as law. The site also 


provided links for students to Career Services. 
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Figure 4. Screen shot of new Arts and Humanities Division website 

Arts & Humanities Division ^ 



The updated website also eorresponded with the redevelopment of a more user-friendly eourse 
catalog website. Previously, students had to weed through scores of class descriptions and could 
become easily overwhelmed and/or overlook similar courses. On this new site, students could 
sort humanities courses of interest by topic. Outreach extended beyond the website to include 
postcards, posters, and a new logo to help launch the framework courses. There is also a new 
freshmen-only website that invites new students to explore more about the Division. 


Additionally, a student advisory committee composed of approximately eight 
sophomores was called together to dialogue with the dean about the humanities on campus. 
These monthly meetings allow the dean to test ideas and get a sense of current perceptions of 
ongoing efforts. The advisory board, for instance, shaped the communication strategy: Members 
discussed how they felt bombarded with social media and felt an alternative form of outreach 
would be refreshing. This feedback helped shape the Division’s website to evoke feelings of 


serenity. 
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The committee made several other recommendations, including adding faculty positions 
and lending greater consideration to co-taught courses across divisions and schools. There have 
also been events. In May, President Faust hosted a public forum. The Humanities and the Future 
of the University, which featured faculty panelists from University of California, Los Angeles, 
Columbia University, and Tufts University. 

Looking forward. Overall, individuals involved in the effort view change as slow, as the 
initiative requires a change in the institutional culture and more broadly, changes in societal 
perceptions of the humanities. Circumstances necessitate the Division to be more responsive to 
the external environment and for faculty to break from the siloed, solitary nature of academic 
research that many are accustomed to. Faculty must look beyond their own research, classes, and 
departments to help solve problems as a Division. 

The current focus of the committee is to continue outreach to departments to obtain 
greater institutional support and to enhance student interest in the humanities. Members of the 
committees have also continued to discuss ways to further the efforts, including developing 
thematically based course clusters that allow cohorts of students to explore topics of 
interdisciplinary nature over the course of two years. A year-long framework course will be 
launched in 2014-2015. Additionally, ideas have been raised about creating spaces on campus 
such as cafes and exhibitions where students can gather and discuss relevant topics. While 
administrators consider efforts to be successful if there is enhanced discussion and awareness of 
the importance of humanities, course enrollments, course evaluations, and numbers of 
concentrators are also being tracked carefully. 


Discussion 
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Although much attention has focused on the declining proportion of bachelor’s degrees in 
humanities, the review of national data offers clarity on the so-called “humanities crisis.” From a 
longitudinal perspective, the height of degree conferrals in the humanities occurred in the 1960s 
during a time when participation in higher education was expanding among women and adults. 
The data suggest that degree conferrals in the humanities have remained relatively stable over the 
past few decades when taking student demographics into consideration. An examination of data 
from the past ten years reinforces arguments that there have not been large shifts in the 
distribution of bachelor’s degree conferrals for most fields. Rather, there have been fields such as 
healthcare that have experienced strong growth, as well as uneven change across career-friendly 
fields (i.e., education) and within fields when looking at individual majors (i.e., English; Visual 
and Performing Arts). The data also suggest that certain institutions, such as the Ivies and the 
Top 50 experienced more acute drops in humanities’ degree conferrals. Based on current trends, 
humanities is unlikely to be a field of tremendous growth, especially when compared to other 
fields such as healthcare and natural sciences. 

The Harvard Humanities Project highlights how declining enrollments became an 
opportunity for innovation within a division and an opportunity for greater cross-departmental 
and interdisciplinary collaboration. While it is early to know the outcomes of the project, the 
initiative represented a broader signal to students and the public at-large about the importance of 
humanities. Further, the project underscored the importance of institutions taking stock of their 
students’ perspectives in every aspect related to course and major selection, from designing 
websites to outreach efforts and new course development. 

The Project fits in with broader discussion about integrating the humanities with real- 
world skill development and experiences. Some institutions have adjusted their curriculum to 
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make direct connections to careers. Stevenson University, for instance, has sought to integrate 
liberal arts education with career development in every course (Markley, 2013). In the higher 
education sector, there has been an emergence of services offered to colleges and universities 
that bridge student learning and career preparation (Stokes, 2013). In 2013, Linkedin launched 
its University Pages, a site designed for high school students to understand the sectors and fields 
alumni with Linkedin profiles enter. Coursesolve also launched in 2013 and is a site that matches 
college courses with companies, organizations, and individuals that seek out postsecondary 
students’ consultation and support to real-world problems. In another example, the Council for 
Aid in Education (CAE) released the Collegiate Eeaming Assessment Plus (CEA-i-), a 
performance assessment which measures students’ critical thinking through problem-solving 
real-world scenarios. Students sift through information in order to generate a solution to a 
problem. The measure assesses students’ scientific and quantitative reasoning, critical reading 
and evaluation, and argument critique. According to the CAE, the CEA-i- can help identify work 
readiness of graduating seniors. 

Additionally, institutions are increasingly finding ways to enhance their career services. 

In a survey of 600 career center directors, the National Association of Colleges and Employers 
(2012) found that only one quarter of directors felt that their students had the right tools to find a 
job. Many career centers are reexamining their offerings and ways they are serving students in 
the challenging economy. A number of institutions have sought alumni assistance in finding 
student or graduate internship placement or employment. York College, for instance, aimed to 
find employment matches for 100 students through its alumni’s employers, and surpassed its 
goals (Harpaz, 2013). Some career programs have humanities majors deliver a personal brand 
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pitch that highlights their employability based on the skills they have developed through their 
studies. 


The debate about humanities’ future remains and there is no shortage of opinions about 
how it must continue to change. Some argue that humanities’ faculty is not doing enough to 
engage students; others have noted that humanities as a scholarly field has been too inward- 
facing and that traditional preparation of humanities faculty are ill-suited to interdisciplinary 
nature of 21**^ century issues. Some scholars see movement toward digital humanities as a 
welcome change. Others point to the need for greater outreach during high school to interest 
students. Certainly, public concern about college graduates’ employment experiences will not 
subside and higher education will need to find ways to adapt. 

Implications for Institutional Researchers 

The institutional case study demonstrates how declining enrollments and changing public 
sentiment about the role of humanities in higher education can lead to opportunities to innovate 
both in the curriculum and in institutional practices. As storytellers of data, institutional 
researchers have the ability to shape policy and institutional innovation (Parmley, 2009) by 
informing leaders about how institutional data fit into the college’s or university’s mission, 
strategy and broader policy discussions. In the case of shifting enrollments, the case study 
highlights the several steps institutional research offices can take in support of committee work, 
as Harvard College’s Office of Institutional Research did with the Humanities Project. 
Institutional researchers can take “deep dives” to identify trends in student enrollment patterns, 
major changes, student satisfaction, and engagement, and help committees and leaders 
understand what the data collectively mean. For instance, institutional researchers can explore 
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questions like: How are enrollments and degree conferrals shifting at one’s institution? How do 
humanities students’ academic experiences compare to their peers from other disciplines? 
Similarly, institutional researchers can explore national and peer data to obtain institutional 
context. 

Additionally, institutional research offices can play an important role in helping leaders 
navigate national policy discussions as Harvard University’s Office of Institutional Research has 
by responding to administrators’ questions and proactively identifying issues. What does the data 
behind national “crises” say? How should we be thinking and talking about some of the 
impending policy changes in higher education? In exploring trends, the findings underscore the 
importance of being mindful of the data parameters — such as the timeline one uses and of 
shifting demographics at one’s institution and nationally — when framing major changes. 

Amidst strong attention toward the humanities and student outcomes, there are many 
opportunities for further exploration of this topic. Future research might examine how 
disciplinary study relates to later life achievements and satisfaction with one’s major. 
Additionally, IPEDS and other national datasets also allow institutional researchers to identify 
how different groups of institutions are experiencing shifts in degree conferrals across fields, 
including liberal arts institutions and non-selective institutions. 
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Abstract 


Web-based surveys help institutions quickly and inexpensively acquire necessary survey 
data, yet plummeting response rates have left us vulnerable to non-response bias. Sending survey 
reminders can be helpful if the newly-recruited participants, albeit late to respond, help us avoid 
this bias. This study examines differences between early, late responders (“procrastinators”), and 
non-responders for two large surveys of undergraduates in two differently-incentivized 
conditions. Procrastinators were characterized by greater dissatisfaction, less engagement with 
faculty, and less engagement with the survey itself. Additionally, late responders in highly 
incentivized situations are likely to be non-responders to low incentive surveys. Analysis 
supports the use of reminders to improve response rate. 



Introduction 


Institutions rely heavily on survey data to achieve planning, assessment, reporting, and 
quality assurance goals (Gonyea, 2005; Porter, 2004b). The advent of web-based surveys has 
allowed institutional researchers to inexpensively distribute surveys to thousands of potential 
respondents with a single click, and to analyze data quickly and easily. Web-based surveys not 
only reduce cost and improve data collection processes, but they afford busy student respondents 
the luxury of completing surveys in comfort and at their convenience. Additionally, many web 
surveys have the capacity to save a student’s progress, allowing for re-entry at a later time 
without sacrificing previously-entered data. Giving students the flexibility to complete surveys 
wherever and whenever they wish assumes that we will get better data when students have time 
to thoughtfully consider each question in a comfortable location. 

However, evidence challenges this assumption: student survey response rates are 
declining, and precipitously (e.g., Dey, 1997; Jans & Roman, 2007; Curtin, Presser, & Singer, 
2005). Because we rely so heavily on data collected from student surveys, plummeting response 
rates are quite concerning. Low response rates often - though not always - result in a sample that 
does not adequately represent the population from which it was drawn (Pike, 2008). Samples that 
are sufficiently dissimilar from the population can lead to non-response bias^ where researchers 
draw erroneous conclusions about the data (Cook, Heath, & Thompson, 2000; Dalecki, 
Whitehead, & Blomquist, 1993). Survey data that represents only vocal minorities are bound to 
be problematic. 

Why might a student fail to respond to surveys? Web surveys might make non-response 
more likely given that emailed invitations and reminders could be accidentally routed to spam 



folders. Or they might encourage forgetfulness - a student who intends to complete a survey at 
another time may forget to follow through. These students are non-responders, but passively so. 
In contrast, active non-responders are students who refuse to do surveys under all but the most 
exceptional of circumstances. 

To combat low response rates, and in turn, poor data quality, we send survey reminders, 
which are as easy to distribute as the original survey invitations. In doing so, we attempt to 
intervene in passive non-responding. In other words, we give the students who are willing to take 
surveys a second chance to follow through. However, are students who need reminders to cue 
their participation different than those who did not need reminders? If so, then ensuring that our 
passive non-responders have ample opportunity to complete surveys is an important strategy in 
avoiding non-response bias. 

Extant research on these late responders - or “procrastinators” - is limited in scope and 
presents mixed findings. These limited findings suggest that late responders are almost always 
male (e.g., Aviv, Zelenski, Rallo, & Larsen, 2002; Stevens & Ash, 2001), and they may be prone 
to troubling behavior, such as low GPA (Bender, 2007; Nielson, Moos, & Lee 1978) or reported 
substance abuse (Kypri, Stephenson & Langley, 2004). Yet a consistent picture of the survey 
procrastinator is still elusive. 

This study addresses three research questions: Are survey procrastinators different from 
those who do surveys right away? If so, how are they different? Linally, how does survey 
incentive impact procrastination? 


Methodology 



Data for this study came from two annual student experience surveys administered to 
sophomores enrolled in the School of Arts and Sciences at Tufts University during the 2012 and 
2013 spring semesters (“Sophomore Survey”). Survey questions are developed by faculty and 
staff to monitor student attitudes and to evaluate academic and campus services. Additional 
demographic and academic data, including gender, race, and GPA, came from the university’s 
data warehouse. 

The 2012 Sophomore Survey was administered to 1151 sophomores. Of this population, 
1073 students began the survey, resulting in a response rate of 93.3%. The survey instrument 
contained 63 items assessing a variety of topics, including satisfaction with advising, 
participation in campus activities and community service, satisfaction with campus services, and 
engagement with faculty. Although the majority of the items on the survey were scored on a 4- 
point or 5-point Likert scales, several items employed a check-all-that-apply, multiple-choice, or 
open-ended format. 

The 2012 Sophomore Survey was highly incentivized. Students who started the survey 
were allowed to begin the registration process for their fall semester classes in advance of the 
university’s registration date. Students choosing not to participate in the survey did not get this 
advantage. Students received an initial email inviting them to participate in the survey. Two 
weeks after receiving the initial email, students received the first reminder; a second reminder 
was issued two weeks after the first. 

The 2013 Sophomore Survey was administered to 1123 sophomores, and was began by 
718 sophomores; the response rate was 64.0%. The survey instrument had been revised 
significantly following a review process and contained 5 1 items assessing a similar range of 



topics and in similar formats as in 2012. Although many items were similar on both surveys, the 
2013 instrument asked fewer questions about advising and more questions about student 
wellbeing. Additionally, the survey was decoupled from registration for the first time; students 
who completed the survey could enter a raffle to win prizes. Students received an initial survey 
invitation, followed by a series of 6 reminders. These reminders were issued at intervals 
averaging 120 hours apart. 

Procrastination was defined as a survey response begun only after a reminder had been 
issued. The time stamp on the data generated by the web-based survey platform was used to 
identify early and late responders. 

Results 

Ad hoc analyses were conducted to test for differences between respondents and non- 
respondents, and then to test for differences between early and late responders. 

2012 Survey 

In the highly incentivized, high response rate 2012 survey, there were no significant 
differences in academic and demographic variables across responders and non-responders. 

Of the responders, 58.7% were categorized as “early responders” by virtue of having 
started the survey prior to the issuance of a reminder. The remaining 41.3% of respondents were 
designated as “survey procrastinators.” Early responders and procrastinators differed 
significantly on two demographic variables and one academic variable. Compared to the sample 
of early responders, more men {'i = 16.91, p < .001) and more nonwhite students {'i = 14.26, p < 



.001) were procrastinators. Additionally, procrastinators had a significantly lower GPA than did 
early responders {t = 7.35, p < .001). 

Procrastinators also differed from non-responders on several survey variables. They 
reported more difficulty choosing a major (t = 2.06, p < .05), they were less likely to make an 
appointment to see their academic advisor (t = 2.22, p < .05), and more likely to drop in to see 
their advisor at the last minute (t = 6.81, p < .01). Procrastinators were less engaged with faculty; 
they were less likely to have asked faculty for a letter of recommendation (x = 9.76, p < .01) or 
for academic advising (x^ = 4.53,p < .05) than early responders. Additionally, procrastinators 
indicated that they were less likely to participate in student organizations (t = 2.99, p <.01) and 
demonstrated more dissatisfaction with their overall Tufts experience compared to early 
responders (t = 1.85, p < .05). Finally, procrastinators took less time to complete the survey than 
did early responders (t = 2.39, p < .05). Tables 1 and 2 display the relevant demographic, 
academic, and survey variable means, and results of chi-square and t-test analyses for the 2012 
survey. 

2013 Survey 

In the low incentive 2013 survey, there were more male non-responders than there were 
male responders (% = 34.70, p < .001). Additionally, the mean GPA for non-responders was 
lower than that of responders (t = 4.20, p < .001). No significant differences in racial 
composition between the groups were observed. 

Of the responders, 48.7% started the survey before the first reminder. There were no 
significant differences between early responders and procrastinators on the two demographic 
variables (gender and race) or on GPA. Procrastinators did, however, differ from early 



responders on a number of survey variables. They reported being less likely to ask a faculty 
member for a letter of recommendation (% = 4.69, p < .05) or for educational opportunities 
outside of Tufts = 4.55, p < .05). They were less satisfied than early responders with the sense 
of community on campus (t = -2.24, p < .05) or where they lived (t = -3. 10, p < .01), and were 
less likely to have had discussions with other students about academic topics {t = 2.12, p < .05). 
Additionally, when asked if they felt they were able to contribute to Tufts more as sophomores 
than as First Year students, non-responders were less likely to say “yes” (x = 4.25, p < .05). 
Finally, procrastinators took less time to complete the survey (t = 3.16, p < .01) and were less 
likely than early responders to leave a comment in the final open-ended question box (“Please 
use the space below to provide any additional comments”; x = 4.12, p < .05). Tables 3 and 4 
display the relevant demographic, academic, and survey variable means, and results of chi- 
square and t-test analyses for the 2013 survey. 

Discussion 

In this study, survey procrastinators - those who wait for reminders before starting a 
survey - were different from those who choose to participate in surveys earlier. In both the 2012 
and 2013 Sophomore Survey, procrastinators were less engaged with faculty, more dissatisfied. 
Additionally, they were less engaged with the survey process overall as evidenced by the amount 
of time spent answering questions and, in 2013, the willingness to leave a comment in the final 
open-ended question box. 

Although the 2012 responders were not different from non -responders for gender, race, 
and GPA, these two groups were indeed different in the 2013 sample. Consistent with previous 
research, non-responders to the low incentive version of the Sophomore Survey were more likely 



to be male and to have a lower GPA than responders. Additionally, this profile also described the 
late responders in the high incentive survey version. Thus, male students at Tufts with a lower 
GPA need a strong incentive in order to gain their participation; else they are non-responders. 

Survey reminders are indeed valuable. As evidenced by the low incentive 2013 survey, 
where less than half the sample started the survey before the first reminder was issued (and 
necessitating six reminders to achieve a 64% response rate), reminders can have significant 
impact. Even in the high incentive condition, more than 40% of survey respondents began the 
survey only after receiving a reminder. Data provided by late responders allowed for a fuller 
picture of the sophomore experience - a picture that would have been much rosier had no 
reminders been issued. 

Although reminders can boost the response rate of a survey, they do not fully combat the 
risk of nonresponse bias. Non-responders in the low-incentive survey resembled the late 
responders in the high-incentive survey, indicating that a strong incentive can turn non- 
responders into responders, albeit late ones. Still, even with a strong incentive, nearly 7% of the 
class did not participate in the survey — these students are the “active non-responders” described 
earlier who will not participate in surveys. We must accept that some portion of the population 
will actively non-respond and account for this when planning survey administration. 

Finally, this study indicated that even a 64% response rate - considered strong in many 
contexts - still places data at risk for nonresponse bias. Our 2013 survey overrepresented women 
and students with higher GPAs, which may have implications for the conclusions drawn from the 


data. 



Several limitations to this study should be noted. First, Tufts University undergraduates 
are traditional students, typically ages 18-22 and completing their bachelor’s degrees in 4 years. 
The findings that late responders are less academically engaged and more dissatisfied may not be 
generalizable to students at all institutions. Additionally, other variables that were not analyzed, 
such as pre-matriculation traits, content of open-ended questions, or analysis of student majors. 
Finally, although we know who the survey non-responders at Tufts are likely to be, we know 
very little about why students choose not to participate in surveys. 
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Table 1 


Demographic and survey variables for 2012 early responders and procrastinators 



Early 

responders 

Procrastinators 



n 

% 

n 

% 

x'(i) 

Male survey participants 

232 

39.6 

208 

52.9 

16.91*** 

Nonwhite students 

147 

24.4 

146 

35.4 

14.26*** 

Have asked faculty for a 
recommendation 

327 

53.5 

186 

43.7 

9.76** 

Have asked faculty for help with 
academic decision making 

469 

76.8 

302 

70.9 

4.53* 


* p < .05 ** p < .01 



Table 2 


Academic and survey variable means for 2012 early responders and procrastinators 



Early responders 

Procrastinators 




M 

SD 

M 

SD 

t 

P 

Mean GPA 

3.48 

0.33 

3.31 

0.40 

7.35 

<0.001 

Number of minutes to 
complete survey 

134.7 

479.7 

90.9 

297.3 

2.39 

0.017 

How difficult was it for you to 
choose a major? a 

2.50 

1.16 

2.64 

1.15 

-2.01 

0.045 

I generally made an 
appointment to see my pre- 
major advisor b 

3.17 

0.75 

3.07 

0.75 

2.21 

0.027 

I usually stopped to see my 
pre-major advisor at the last 
minute b 

1.93 

0.75 

2.26 

0.84 

-6.81 

<0.001 

How often have you 
participated in activities 
sponsored by a student 
organization? c 

3.10 

0.89 

2.93 

0.95 

2.98 

0.003 

If you had the chance to relive 
the college choice process, 
would you choose to attend 
Tufts again? d 

4.01 

1.01 

3.90 

1.02 

1.85 

0.049 


a The scale used for this item was: 1 = Very easy, 2 = Easy, 3 = Neither easy nor difficult, 4 = Difficult, 5 
= Very difficult 

b The scale used for this item was: 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree 

c The scale used for this item was: 1 = Never, 2 = Occasionally, 3 = Often, 4 = Very often 

d The scale used for this item was: 1 = Definitely not, 2 = Prohahly not, 3 = Mayhe, 4 = Prohahly, 5 = 
Definitely 



Table 4 


Demographic and survey variables for 2013 early responders and procrastinators 



Early 

responders 

Procrastinators 



n 

% 

n 

% 

x'(i) 

Male survey participants 

128 

36.6 

152 

41.3 

1.69 

Nonwhite students 

93 

30.8 

91 

29.7 

0.80 

Have asked faculty for a 
recommendation 

205 

63.1 

175 

54.7 

4.69* 

Have asked faculty for additional 
educational opportunities outside 
of Tufts 

128 

39.5 

101 

31.5 

4.55* 

Do you feel you are able to 
contribute more to the Tufts 
community now than as a First 
Year student? (% yes) 

295 

51.3 

280 

48.7 

4.25* 

Wrote in final comment box (% 
yes) 

98 

28.0 

79 

21.5 

4.12* 


* p < .05 



Table 4 


Academic and survey variable means for 2013 early responders and procrastinators 



Early 

responders 

Procrastinators 




M 

SD 

M 

SD 

t 

P 

Mean GPA 

3.45 

0.37 

3.43 

0.38 

0.84 

0.402 

Number of minutes to complete 
survey 

40.18 

68.54 

26.17 

40.46 

3.16 

0.002 

How satisfied are you with the 
sense of community on campus? a 

2.75 

0.75 

2.88 

0.89 

-2.24 

0.025 

How satisfied are you with the 
sense of community where you 
live? a 

2.69 

0.89 

2.90 

0.82 

-3.10 

0.002 

How often have you had 
discussions with students about 
academic topics? b 

3.49 

0.64 

3.38 

0.69 

2.12 

0.035 


a The scale used for this item was: 1 = Very dissatisfied, 2 = Dissatisfied, 3 = Satisfied, 4 = Very satisfied 


b The scale used for this item was: 1 = Never, 2 = Occasionally, 3 = Often, 4 = Very often 
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Introduction 

Student retention has been increasingly recognized as a critical issue by American colleges and 
universities since the early 1970s [1]. Retention is directly related to student educational 
attainment and time to degree completion. Therefore, graduation rates are often used by the 
public and government to measure the success and effectiveness of higher education institutions 
[2, 3]. A number of previous studies have focused on the persistence and graduation rates of 
underrepresented minority students, students with lower socioeconomic status, and first- 
generation students [4-8]. However, there are not many studies quantifying the retention and 
graduation performance for freshmen with aspirations to obtain their degree in a STEM (Science, 
Technology, Engineering and Mathematics) discipline, despite the fact that STEM has become 
one of the progressively hot topics inside higher education and for funding initiatives in recent 
years. According to a 2010 University of California- Los Angeles study [9], there is a substantial 
number of undergraduates across the country choosing to leave STEM programs before they 
graduate with a STEM degree, and many students who start in those STEM programs struggle to 
finish their degree within four years, or drop out. This loss after college admission is critical 
given the efforts to enhance STEM enrollment due to the growing demands of a highly skilled 
workforce and the shortage of STEM graduates production from our colleges and universities 


[ 10 , 11 ]. 



Previous studies on retention have found that a student’s decision to remain at an institution is 


due to personal characteristics, academic background, and integration into the academic and 
social life of the campus [12]. A number of researchers have linked academic ability and 
achievements with students’ persistence in college [13]. In addition to the background of 
students, the characteristics of the institution are also relevant to students’ persistence and 
success. Adequate financial aid, individual academic support systems, better social and cultural 
support systems, and a welcoming campus environment were also found to promote retention. 
Whether these risk factors from traditional attrition models also play a role in students’ decision 
to change their majors from STEM to Non-STEM have not been fully understood. 

Methodology 

The Consortium for Student Retention Data Exchange (CSRDE) at the University of Oklahoma 
has expanded the main CSRDE survey to include an optional survey which summarizes and 
benchmarks the retention and graduation rates of first-time full-time freshman cohorts majoring 
in STEM. In our study, data from the CSRDE main and STEM surveys for the 2007-201 1 
cohorts are used to compare first to second year retention rates of the entire entering cohorts of 
first-time full-time freshmen: institution- wide and discipline-specific for STEM students of the 
same entering cohort. 

The analysis uses data from five historical fall entering first-time full-time freshman cohorts, 
2007 to 201 1. Depending on which detailed retention and graduation measurements are being 
compared, different entering cohorts are used for rate calculations. Eor example, for the second 
year retention rates, all five cohorts can be used. However, the four-year graduation rate can only 
be generated for the 2007 and 2008 cohorts. 



The CSRDE STEM survey defines which disciplines are to be considered STEM and uses the 
2010 CIP codes for further definition. The 2010 CIP codes representing STEM disciplines 
include 03. XXXX, ll.XXXX, 14.XXXX, 15.XXXX, 26.XXXX, 27.XXXX, 40.XXXX. 
Additionally, it also includes 01.0000, 01.0801, 01.09 through 01.9999 and 30.1901. According 
to the CSRDE STEM survey definitions, a total of 87 academic plans at the University of 
Delaware are flagged as STEM disciplines, among which more than 50% are from the College of 
Engineering or the College of Arts and Sciences. 

In general, retention outcome is whether a first-time full-time freshman is retained from the first 
fall to the second fall semester. STEM institution- wide retention outcome is reflecting whether 
entering students majoring in a STEM discipline are retained in any major (STEM or Non-STEM) 
in the following fall semester. In contrast, STEM discipline specific retention outcome is 
measuring whether entering students majoring in a STEM discipline are persisting in any one of 
the STEM majors. Eor this latter outcome, either students who didn’t enroll or students enrolled 
in non-STEM majors are regarded as non-retained under the discipline specific retention 
definition. 

The four- year graduation outcome is the primary graduation outcome of interest. Similar to the 
retention rates, the three types of graduation rates are representing (1) the percentage of first-time 
full-time freshmen who completed their degree in four years; (2) the percentage of STEM 
entering students who completed their degrees in any field in four years; and (3) the percentage 
of STEM entering students who completed their degree in one of the STEM disciplines in four 
years. In the third outcome, if a STEM entering student fails to graduate before the 5* fall with a 



STEM degree they are regarded as non-graduated under the discipline specific graduation 
definition. 

The percentage descriptive statistics are calculated to compare the three types of retention rates 
and graduation rates overall, by gender, and by race/ethnicity. Multinomial logistic regression 
analyses were performed among the STEM entering students to evaluate the potential risk factors 
relating to increased partial odds of changing to Non-STEM major as opposed to persisting in 
STEM programs from the first fall to the second fall. The STEM entering students in all five 
entering cohorts are used for the analyses. Eactors being evaluated include demographic factors 
(gender, race, age, residency), socioeconomic factors (low income. Pell Grant, first generation), 
academic background (honor student, SAT scores), and which college initially enrolled in. Odds 
ratios (OR) after adjusting for cohort effect and 95% confidence intervals (Cl) are used as 
indicators of the strength of association. A p-value less than 0.05 is considered as statistically 
significant throughout the study. 

Results 

The sample population from the past five entering cohorts is 18,143, among which 4,873 (26.9%) 
students are initially majoring in a STEM discipline. The sample size of the five cohorts is 
consistent, ranging from 3,365 (2010 cohort) to 3,905 (2011 cohort). Over the past five years, the 
overall proportion of STEM students in the entering cohorts has been relatively stable, 
fluctuating between 24.3% and 27.6%. Male students far outpace their female peers majoring in 
a STEM discipline throughout the entire five entering cohorts. There is a slightly upward trend 
for the Underrepresented Minority group (URM) starting with a STEM major from 2007 to 2012 



(Figure 1), which is likely the result of recent STEM recruitment initiatives or the increasing 
number of Non-Resident Aliens at the University of Delaware. 



Figures 2 to 4 show the comparison of three types of Yearl - to-Year2 retention rates by cohort 
only, by cohort and gender, and by cohort and race (only 2007 data are presented). The retention 
rates for the entire five entering cohorts of first-time full-time freshmen are consistently above 90% 
at UD. There is no doubt that the STEM entering students, if not outperform, performed as well 
as well as overall first-Time full-time freshmen with regard to the Institution-wide retention rates. 
However, when comparing that with the discipline specific retention rates, the retention rates for 
STEM entering students drops by an average of 13% across the five cohorts, indicating 
essentially 13% additional STEM-specific attrition among those STEM entering students (Figure 
2). Although females tend to have slightly higher retention rates for overall first-time full-time 
freshmen and higher institution-wide retention rates for STEM entering students, they invariably 
lag behind males with respect to STEM discipline-specific retention performance across all five 


cohorts (Figure 3). Both African-Americans and Hispanics are subject to the highest proportion 


(18%) of STEM-specific attrition from the first fall to the second fall semester. 


Figure2 First yr to second yr retention rate, retention rate for overail FTIC-FT students 
vs. institution -Wide retention rate for STEM entering FTiC-FT students vs.Discipiine- 
Specific retention rate for STEM entering FTiC-FT students 
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Figure4 First year to second year retention rate by cohort term and race, retention rate 
for overail FTIC-FT vs. institution -Wide retention rate for STEM enteringStudents vs. 
Discipline-Specific retention rate for STEM entering Students 
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Approximately 68% of the entire five cohorts of first-time full-time freshmen graduated within 
four years. A similar proportion of STEM entering students graduated from any major within 
four years, while only close to 50% of them graduated with a STEM degree within four years 
(Eigure 5). Similar to retention rates, in comparison to males, females have higher four year 
graduation rates for one of the two cohorts of first-time full-time freshmen and the subset of 
STEM entering students, if not taking into consideration what degree they completed. In contrast, 
STEM entering males outpace their female peers in the percentage of obtaining a STEM degree 
within four years (Eigure 6). There is great variation in the four year graduation rates for 
different race/ethnicity groups. African-Americans are far behind others in all three types of 
measurements for four- year graduation rates. STEM entering Hispanics are likely to have higher 


four- year graduation rates graduating from any major than the overall Hispanic subpopulation. 


STEM entering Non-Resident Aliens seem to have the highest proportion of graduates with a 


STEM degree, but their overall graduate rates are less than 50% (Eigure 7). 


Figures 4yr Graduation rate by cohort term, graduation rate for overali FTiC FT 
students vs. institution -Wide graduation rate for STEM entering FTiC-FT vs. 
Discipiine-Specific graduation rate for FTiC FT students 
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A total of 4,488 (92.1%) out of 4,873 STEM entering first-time full-time freshmen persist in the 
second year, among which 3,866 (86.1%) continue to major in one of the STEM programs at UD. 
Individual factors significantly associated with higher risk for changing to Non-STEM majors as 
opposed to persisting in a STEM program include being a female, being a Delaware resident, 
being the first person in their family to attend college, having a SAT math score less than 650; 
while Non-Resident Aliens/Intemational students. Honors students, or students who started their 
STEM program in the College of Engineering seem less likely to switch their major to non- 
STEM (Table 1). 

Although STEM-specific attrition is not a favorable outcome, it may indicate whether students 
were able to receive sufficient academic advising when they realized a STEM major was not a 
good fit. In that sense, female students, Delaware residents, and Honors students are more likely 



to change to non-STEM programs relative to dropping out. However, International students and 
students with an initial STEM program in the College of Engineering are more likely to drop out 
as opposed to change to a non-STEM major (Tablel). 


Tablet Relationship between individual factors and the likelihood of changing to Non-STEM majors 


Factors 


Partial Odds Ratio* (95% Confidence Interval) 

out of STEM vs. persist in 
STEM 

out of STEM vs. 
discontinue 

Gender 

Female vs. Male 

2.2(1.9,2.6) 

2.1(1.6,2.7) 


Asian vs. White 

1. 0(0.7, 1.4) 

1.1 (0.6, 1.8) 


Black vs. White 

1.2(0.8,1.8) 

0.6(0.4,1.1) 

Race/Ethnicity 

Hispanic vs. White 

1. 0(0.7, 1.4) 

0.7(0 .4, 1.3) 


NR A vs. White 

0.4(0.2,0.7) 

0.3(0.1,0.6) 


Other' vs. White 

0.7(0.4,1.2) 

0.7(0 .4, 1.4) 

Underrepresented 

Yes vs. No 

1.1 (0.8, 1.4) 

0.7(0.5,1.0) 

International Student 

Yes vs. No 

0.4(0.2,0.7) 

0.3(0.1,0.6) 

Residency 

in-state vs. out of state 

1.3(1. 1,1.5) 

1.7(1.3,2.3) 

Pell grant receiver 

Yes vs. No 

1.3(1.0,1.7) 

0.7(0.5,1.0) 

First generation 

Yes vs. No 

1.4(1. 1,1. 8) 

0.9(0.6,1.2) 

Low income 

Yes vs. No 

1.1 (0.7, 1.7) 

0.6(0.4,1.1) 

Honor student 

Yes vs. No 

0.7(0.5,0.8) 

1.7(1.2,2.4) 

SAT Math score 

<650 vs. >=650 

2.1(1.8,2.5) 

1. 2(0.9, 1.5) 

SAT Reading score 

<600 vs. >600 

1.1(0.9,1.4) 

0.8(0.6,1.0) 


Agriculture vs. Engineering 

1.9(1.3,2.7) 

1.9(1. 1,3.2) 

College originally enrolled 

Arts&Sciences vs. Engineering 

3.4(2.8,4.2) 

3.0(2.3,4.0) 

in 

Health Science vs. Engineering 

13.1(6.2,27.6) 

3.4(1. 2,9.6) 


Earth Ocean vs. Engineering 

2.5(1.7,3.8) 

3.3(1.6,6.8) 


Other includes race/ethnicity unknown, multi-ethnic, American Indians, Native Hawaiians and Pacific Islanders 


* All odds ratios were controlled for the cohort effect. 


Conclusion/Discussion 


Without question, science and engineering capability will be the foundation of economic success 
for the U.S. in the 21st century. According to the U.S. Bureau of Statistics, in the next five years, 
STEM jobs are projected to grow twice as fast as jobs in other fields. Based on this projection, 
the U.S. will have over 1 million job openings in STEM-related fields by 2018. Yet only 16% of 
graduates in U.S. will specialize in STEM (data from U.S. Bureau of Statistics). Apparently, our 
education system is not preparing enough STEM majors to meet the demand. The results of this 
study visually underscore the leak in the STEM education pipeline for higher education. The 
gender and racial differences in STEM retention rates and graduation outcomes are highlighted 
as well. This study calls for the need to regularly track STEM discipline-specific retention and 
graduation rates to raise concerns and attention among senior administrators, especially for 
institutions with impressive overall retention and graduation rates. 

Students’ personal characteristics and academic background have significant impact on their 
decisions to persist in STEM programs. It is interesting to know that students who begin a STEM 
program in the College of Engineering are less likely to change to a non-STEM program, 
compared to other STEM entering students. This may be due to the fact that males are largely 
overrepresented in the College of Engineering or the College is providing better academic 
support services. Eemales still lag behind in representation of both the STEM entering and 
STEM graduating populations. Compared to their male peers, STEM entering females are 
subject to more STEM specific attrition after college admission, although they are less likely to 
drop out from the University. There are multiple theories to explain the gender gap including the 
test based theories, biological determination theories, cognitive learning difference theories, and 



social-psychological theories [14] .Future studies focusing on assessing students’ attitudes and 

beliefs about women in STEM-related disciplines will be helpful to better understand the loss of 

female STEM graduates. 
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Abstract 


Benchmarking, the process of comparing an institution's performance against other 
institutions, provides unique opportunities to identify strengths and weaknesses. In this paper, 
we discuss the value benchmarking brings to program review and how Cambridge College has 
used benchmarking information to produce actionable data. We discuss how we shared that 
information with decision-makers and faculty/staff, leveraged the peer benchmarks to gain 
additional resources, improvements in student learning outcomes, lessons learned, and future 
plans. 

Introduction 

Program review is a process of determining the current state of a program by providing 
credible evidence collected from key stakeholders (e.g., students, faculty, administrators, alumni, 
employers). Conducting a program review requires the development of a systematic method of 
analyzing components of an instructional program, including instructional practices, aligned and 
enacted curriculum, student work samples, formative and summative assessments, professional 
development and support services, and administrative support and monitoring. A critical 
component of program review is assessment of student learning outcomes. Palomba and Banta 
(1999) wrote that assessment is the systematic collection, review, and use of information about 
educational programs undertaken to improve student learning and development. 

Institutions assess programs for many reasons namely to provide evidence of effectiveness 
for accreditation or performance funding and to justify resource allocations, but most 
importantly, to understand their program’s strengths and weaknesses in order to improve their 
programs. Strong academic and student affairs programs lead to strong student outcomes. 



Creating Effective Assessments 

Effective assessments provide action-driven information regarding students’ experiences 
with that program. There are three basic components to an effective assessment: 

• Valuable information: The student experience is broad and program evaluations should 
include environmental and associated student learning outcomes. Environmental metrics 
could include academic advising, facilities, quality of instruction, and peer connections. 
Student learning outcomes should align with appropriate accreditation or professional 
standards. 

• Reliable and valid instruments: Bad survey instruments leads to bad data; making 

decisions from bad data costs institutions valuable resources. Reliability and validity 
testing (e.g., Cronbach’s alpha, test-retest, and construct validity) must be used to ensure 
a credible instrument is used. 

• Action-based reporting: Reporting must provide information of strengths and 

weaknesses, changes over time, and focus the program on areas on needed improvement. 
Benchmarking provides needed perspective. 

Benchmarking 

Benchmarking is the process of comparing one’s program to others’ programs using defined 
metrics. Eor most institutions, the desire and willingness to learn from each other is grounded in 
collegiality and the commitment to improving the state of higher education as a whole (Bender, 
2002). Jackson and Eund (2000) write that collaborative benchmarking in higher education is 



aimed at people who have a responsibility for evaluating institutional policies, practices, and 


performance. 

Most regional and program accreditors require assessment information in support of current 
processes and encourages the inclusion of external benchmarks. For example, the New England 
Association of Schools and Colleges (NEASC) Standard 4.54 reads '"The institution uses a 
variety of quantitative and qualitative methods and direct and indirect measures to understand 
the experiences and learning outcomes of its students, and includes external perspectives. ” 

Benchmarking adds perspective to an assessment project. An external view provides 
institutional leaders comparative data about their programs and helps answer the ultimate 
questions “Are we preforming well?” or “Is it possible to perform better?” or “We are 
performing at the top of our profession!”. 

EBI has created various assessment tools based on the premise that assessment is vital to 
continuously improving programs, especially in the area of teacher preparation education. One 
particular instrument, the Teacher Education Exit Assessment, was designed to benchmark 
assessments based on the NCATE' standards for program accreditation. The purpose of the 
Teacher Education Exit Assessment is to provide institutions insight into their program based on 
national standards and to target meaningful changes backed by hard data to support institutional 
goals of continuous improvement and accreditation. There are three benchmarking groups as 
part of the reporting function: 

• Select 6 / Peer Institutions: A comparison of institutional results against four to six peer 
or aspirant institutions. 


* NCATE and TEAC merged this past summer into CAEP. CAEP advances excellence in 
educator preparation through evidence-based accreditation that assures quality and supports 
continuous improvement to strengthen P-12 student learning. 



• Carnegie Classification Institutions: A comparison of institutional results against the 
institutions in the same Carnegie Class. The Carnegie Classification of Institutions of Higher 
Education is a framework for classifying, or grouping, colleges and universities in the United 
States. EBI uses the 2010 Basic Carnegie Classifications to determine an institution’s 
classification. All institutions outside the United States are combined into an “International” 
Carnegie Class. 

• All Institutions: A comparison of institutional results against all participating institutions. 
This provides a national norm or national standard. 

EBI Teacher Education Exit Assessment 

The EBI Teacher Education Exit Assessment measures the effectiveness of the teacher 
preparation program from the students’ viewpoint. Over 100 colleges and universities have 
surveyed over 150,000 students since that assessment began in 1999. Demographic questions 
include gender, ethnicity, and race and additional categorical questions relate to future 
employment and future education. Qualitative questions ask the respondent to provide 
suggestions for program improvements. There are 35 seven-point Eikert-scaled questions to 
provide feedback on environmental metrics like the student teaching experience, career services, 
and support services. There are 36 seven-point Eikert-scaled questions to provide measures of 
student learning outcomes that align with CAEP, NCATE, and TEAC specialized accreditation. 
These 71 Eikert-scaled questions collapse into 16 factors with Cronbach’s alpha ranging from 
0.78 to 0.95. 

The principal report provided by EBI is Recommendations for Improvement. This report, the 
product of a multi-variant linear regression, identifies the factors which predict the students’ 
perceptions of their experience with this program. EBI separates the factors into two groups: 



• No/Low Impact Factors: These factors fail to strongly predict students’ pereeptions of the 
program. Applying additional resources to improve these areas will not improve 
students’ perceptions of the program. 

• High Impact Factors: Focusing additional resources (or reallocating other resources) to 
improve these faetors will result in an overall improvement in students’ pereeptions of 
the program. 

Cambridge College has participated in the EBI Teacher Education Exit Assessment for three 
years. Below we discuss how we applied those results for program improvement. 

Applying Assessment Results for Improvement 

Cambridge College was established by a small group of educators who created an 
educational institution whose primary mission was to provide educational opportunities and 
degrees for adult learners who did not have ready access to higher education. Bounded first as a 
teacher’s college it has grown in size and seope. Cambridge College's School of Education is 
broadly based, inclusive, and committed to producing quality teachers that become agents of 
change in their schools. It provides students with the knowledge, skills, and values to excel 
academically and professionally. Improving students’ perceptions of our program is very 
important. 

The College collects data, such as grades, student teaching evaluations, and state licensure 
test results, because of the requirements for approval by the Massachusetts Department of 
Elementary and Secondary Education (DESE). The College also administers the EBI Teacher 
Education Exit Assessment to graduates of the School of Education. Students are asked to 


evaluate their experience at Cambridge College in categories, such as Satisfaction with Eaculty 


and Courses, Classroom Equity and Diversity, and Teaching Pedagogy and Techniques. 
Questions from the EBI have been mapped to program specific outcomes and triangulated with 
data from other assessment tools. 

As part of its efforts to improve its teacher preparation programs the College embarked on 
seeking national accreditation through TEAC. TEAC accreditation, in part, requires the 
submission of an auditable research brief followed up by a site audit. The research brief was 
focused on establishing a valid and reliable program review process. The incorporation of 
external perspectives and the use of benchmarking were critical components of the brief. The 
review of evidence and auditing revealed that the College and its School of Education require 
changes. Eurthermore, the TEAC process has made clear that the College must maintain 
momentum and continue its development of a robust, accurate assessment system that imbues 
student- learning outcomes. 

TEAC Claims 

The TEAC brief starts with the College faculty stating claims regarding student-learning 
outcomes. The claims are: 

Claim 1 

Cambridge College Teacher Preparation graduates demonstrate content knowledge in 
their fields of Licensure. 

Evidence of Content Knowledge 

Claim 2 

Cambridge College Teacher Preparation graduates demonstrate pedagogical content 
knowledge and teaching skills that promote student learning. 

Evidence of teaching skills and pedagogical knowledge 

Claim 3 

Cambridge College Teacher Preparation graduates are diverse, caring adult educators 
trained to meet the educational needs of their students. 

Evidence of diverse and caring educators 


Mapping the EBI and Using Data for Continuous Improvement 

As an online survey Cambridge College administers to graduates of the Sehool of Edueation, 
the EBI. The survey asks students to evaluate their experience at Cambridge College in 
categories, such as Satisfaction with Eaculty and Courses, Classroom Equity and Diversity, and 
Teaching Pedagogy and Techniques. As mentioned previously, the survey uses a l-to-7 Eikert 
scale wherebyl equals strong disagreement/very poor and 7, strong agreement/exceptional. The 
mean response for each question is then reported as a percentage performance in which 1, 4, and 
7 equate with 0%, 50%, and 100% performance, respectively. 

The EBI survey sets a goal for institutional respondent performance of 75% or higher. 
Additional close-ended questions, such as gender, race, ethnicity, and plans after graduation, are 
also asked to provide demographic background. Each scaled question is reported with reference 
to the background information of the student respondents for that particular question. The 
College uses the measures of peer institution comparisons through Select 6, Carnegie Class, and 
All Institutions rankings. The Select 6 used by the College in 2010-2011, were American 
International College, McKendree University, Metropolitan College of NY, University of 
Wisconsin-Stout, and Eort Hays State University. 

In 201 1 -20 1 2 Cambridge College selected McKendree University, Metropolitan College of 
NY, Eort Hays State University, and Oral Roberts University. The Carnegie Class consisted of 
EBI-designated institutions within the same tier as Cambridge College whose performance is 
reported in aggregate. Thirteen institutions were reported in 2010-201 1 and seventeen in 2011- 
2012. We also used the EBI analysis and benchmarking against all institutions that participated 
in the survey. Cambridge College’s responses were compared to forty-four institutions in 2010- 


2011 and thirty-four in 201 1-2012. 


We use the EBI to study graduates’ satisfaction with our teacher preparation programs as 


well as mapped its questions to state standards and the TEAC Quality Principles and Cross- 
Cutting Themes. 


Categories 

Correspondence: State Standards 

CoiTespondence: 

TEAC 

Subject 

Matter/Pedagogic al 
Knowledge 

A: Plans Curriculum and Instruction 
B: Delivers Effective Instruction 



C: Manages Classroom Climate and 
Operations 


Meets Professional 
Responsibilities 

D: Promotes Equity 
E: Meets Professional 

Responsibilities 

Cross-Cutting Themes 
of Teaming How to 
Team, Technology, 
and Multicultural 
Accuracy 

Satisfaction with 
Program Components 
and Cambridge College 
Administration 


Third Quality 
Principle 

The questions provide feedback about the quality of instruction and candidate support. 


Results of the EBI Benchmarking to TEAC Claims 

In 2010-201 1 the survey response rate (the number of responses divided by the number of 
attempted responses) for Cambridge College was 48.4% (n2on=316, N=652). The overall 
number of graduates in the K-12 licensure program was more than 43% (n=281). In 201 1-20I2 
the responses rate was 39.4% (n=320, N=808). The overall number of completers in the K-12 
licensure program during this period was more than 29% (n2oi2=289). 

Generally, the highest rates reported by graduates in both years were with the quality of 
faculty and courses (in 2012 the scores for performance of 86.7% satisfaction with a high-score 
of 90.7% for average class size). They also report high scores with the interactions they had with 



fellow students in the program (86.7%) and Classroom Equity and Diversity (80.5%). As an 
institution founded on the principles of andragogy this is a critical learning outcome in the 
program^. 

Our EBI analysis identifies two topics for the College to focus its time, energy, and 
resources: Impact on Overall Program Satisfaction and Eevel of Program Satisfaction. Impact on 
Overall Satisfaction indicates the degree to which the factor, if improved, will increase. Overall 
Satisfaction. Improvement in high impact factors will have the greatest affect on Overall 
Satisfaction. Eevel of Satisfaction describes the degree of satisfaction students reported with a 
particular factor: the lower its level, the greater the opportunity for improvement. The most 
efficient and effective way to improve program quality is to concentrate on improving factors 
likely to have greatest impact and factors due to the lowest performance. The high impact factors 
identified by the EBI analysis include: 


Item 

Contribution to 


Total: % 

Quality of Instruction 

30.3 

Student Teaching 

20.9 

Support Services 

18.7 


Although classified as having a no or low impact. Career Services reportedly was the only 
factor that scored significantly below the goal and, therefore, was included in our analysis as a 
low satisfaction indicator. Eurthermore, Overall Program Effectiveness, while not a high impact 
or low satisfaction indicator, was included in our analysis because it provides a holistic 
representation of a student’s experience. 


^ Andragogy is the methods or techniques used to teach adults as advanced by Malcolm Knowles. The process 
focuses on engaging adult learners with the strTicture of learning experience. 



Overall Program Effectiveness 

Overall Program Effectiveness on average achieved a 5.32 average rating (n2oi2=289), falling 
shy of the 5.5 goal but not yet falling into an area a concern (5.25 or lower). When disaggregated 
by M.Ed. Eicensure program concentration, the programs that scored the indicator lowest were: 

• within .25 of goal (5.5): 

o English as a Second Eanguage (5.47, n=86) 

• lower mean than goal (5.5) by more than 2.5: 

o Elementary Teacher (4.62, n=91) 
o Mathematics (4.29, n=26) 
o Health/Eamily & Consumer Service (3.89, n=16) 

Overall Program Effectiveness increased (0.10, n2oi2=289) in 2012 versus 201 land exceeded 
the scores of the Carnegie Class and All EBI Institutions. It scored nearly equal with Cambridge 
College’s Peer Institutions. 

Quality of Instruction 

As the highest impact indicator. Quality of Instruction scored slightly below the factor 
performance with 5.45 (n2oi2=289). Within Quality of Instruction, only the Eeedback on 
Assignments sub-question scored below the goal with an average rating of 5.29 (n2oi2=289). 
When disaggregated by M.Ed. Eicensure program concentration, the programs that scored the 
indicator lowest are: 

• within .25 of goal (5.5): 

o English as a Second Eanguage (5.33, n=86) 
o Special Education/Moderate Disabilities (5.49, n=94) 


• lower mean than goal (5.5) by more than 2.5: 

o Health/Family & Consumer Service (4.42, n=16) 
o Mathematics (4.75, n=26) 
o Elementary Teacher (5.06, n=61) 

Quality of Instruction increased (.03, n2oi2=289) in 2012 versus. 2011 and exceeded the 
scores of the Carnegie Class and All EBI Institutions. It scored nearly equal to Cambridge 
College’s Peer Institutions. Overall, the data suggest that Cambridge College is improving its 
quality of instruction. 

Student Teaching Experience 

As the second-highest impact factor. Student Teaching Experience generally scored above 
the goal for factor performance at (6.17, n2oi2=289). All of the sub-questions within the indicator 
met or exceeded the 5.5 goal. When disaggregated by M.Ed. Eicensure program concentration, 
all programs scored above 5.5. 

Student Teaching Experience remained constant in 2012 versus 2011, increasing by (0.01, 
n2oi2=289). While four of the sub-questions showed an increase in satisfaction between 2011 and 
2012, the remaining questions, process of securing a position, and choice of assignments 
decreased by (0.13, n2oi2=289) and (0.21, n2oi2=289), respectively. Student Teaching Experience 
exceeded the scores of the Carnegie Class and All EBI Institutions. It scored nearly equal with 
Cambridge College’s Peer Institutions. 

The data suggest that, although Student Teaching Experience at Cambridge College is 
excellent, some attention should be given to improving securing a position and the choice of 


assignments. 


Support Services 


As the third high-impact factor, Support Services scored above the goal for factor 
performance at (5.64, n2oi2=289). Within Support Services, only the Training to Utilize 
Education School’s Computing Resources sub-question scored below the goal with an average 
rating of (5.46, n2oi2=289). 

When disaggregated by M.Ed. Licensure program concentration, no programs scored the 
indicator between 5.25 and 5.5. Those programs that scored the indicator lower mean than goal 
(5.5) by more than 2.5 are: 

• Elementary T eacher (4 . 8 8 , n=6 1 ) 

• Health/Eamily & Consumer Service (4.89, n=16) 

• Special Education/Moderate Disabilities (5.12, n=94) 

Support Services decreased by (0.11, n n2oi2=289) in 2012 vs. 2011. The majority of this 
decrease came from the question. Quality of Library Resources, which declined (0.17, n2oi2=289) 
between 2011 and 2012. Support Services exceeded the scores of the Carnegie Class and All EBI 
Institutions. However they fell below the average for Cambridge College’s Peer Institutions. 

The data suggest that, although support services at Cambridge College are generally 
excellent, special attention should be paid to training for the school’s computing resources and 
the quality of library resources, especially the Elementary Teacher, Health/Eamily & Consumer 
Service, and Special Education/Moderate Disabilities programs. 


Career Services 


Although a no or low impact factor, Career Services was the only other to score consistently 
in concern (4.36, n2oi2=289). Within Career Services, all of the sub-questions were average 
scores that fell within the our concern (lower than 5.5 by more than .25): 

• number of schools recruiting on campus (4.00, n2oi2=289) 
quality of schools recruiting on campus (4.02, n2oi2=289) 

• number of interviews had with employers (4.23, n2oi2=289) 

• notice of job openings (4.26, n2oi2=289) 

• assistance in preparing you for your permanent job search (4.72, n2oi2=289) 

When disaggregated by M.Ed. Licensure program concentration, the programs that scored 
the indicator lowest are: 

• within .25 of goal (5.5): 

o Mathematics (5.38, n=26) 
o School Administration (5.40, n=30) 

• lower mean than goal (5.5) by more than 2.5: 

o Elementary T eacher (3 . 3 6 , n=6 1 
o Special Education/Moderate Disabilities (3.84, n=94) 
o English as a Second Language (4.04, n=86) 

Ratings for Career Services decreased by 0.28 (n2oi2=289) in 2012 vs. 2011. By question, the 
largest decrease between 2011 and 2012 is: 

• Quality of schools recruiting on campus (-0.39, n2oi2=289) 

• Number of schools recruiting on campus (-0.31, n2oi2=289) 

• Number of interviews you had with employers (-0.31, n2oi2=289) 



Career Services was on par with the scores of the Carnegie Class and All EBI Institutions. 
However, it fell below the average for Cambridge College’s Peer Institutions. 

While Career Services is not a high impact factor, the low satisfaction rankings make it a 
priority for Cambridge College, which recently hired a career counselor to assist students. 

NCATE Unit and University Teacher Education Program Standards 

In addition to analysis by factor, the EBI results reported on external benchmarks for 
NCATE Unit Standards and Standards for University Teacher Education Programs. Eor all the 
questions reported for NCATE Unit Standards, Cambridge College exceeded its Peer 
Institutions, Carnegie Class, and All EBI Institutions scores. The following questions the 
Program scored below its Peer Institutions: 

• To what degree does your Education course work enhance your ability to effectively 
develop a lesson plan and 

• Availability of Education School’s Computers. 

Similarly, of all of the questions reported for University Teacher Education Program 
Standards, Cambridge College exceeded its Peer Institution, Carnegie Class, and All EBI 
Institutions scores with the exception of the question, “To what degree does your Education 
course work enhance your ability to Effectively develop a lesson plan,” where it scored below 
them. 

Recommendations for Improvement and External Benchmarking 

Through the Recommendations for Improvement report, three factors were identified as areas 
of focus to improve students’ perceptions of our program. 

• U' Predictor - Quality of Instruction: Our program performed well against the institutions 
in our Carnegie Class and all participating institutions but we were statistically equal to 


our peer institutions. The lowest mean question in that factor focused on the quality of 
feedback on assignments. 

• Predictor - Student Teaching Experience: All metrics met or exceed the goal value 
(mean of 5.50 on a 7-point Likert scale) but the lowest item was “How satisfied were you 
with the process of securing a position”? In addition, our program performed well 
against the institutions in our Carnegie Class and all participating institutions but we were 
statistically equal to our peer institutions. 

• Predictor - Support Services: While our program performed well against the 
institutions in our Carnegie Class and all participating institutions, we did not perform 
well in comparison to our peer institutions. The lowest mean question in this factor 
focused on training to utilize students’ use of computing resources. 

Ten factors were listed as not being a strong predictor of students’ perceptions of the 
program. All factors, except for Career Services, met or exceeded the goal value (mean of 5.50 
on a 7-point Likert scale). Career Services was the lowest performing factor in the study. 

Reviewing the external benchmarking results, we found we performed well exceptionally 
well against the three benchmarking groups on the factor “Satisfaction with Laculty and 
Courses” and performed lower-than-desired on two factors: Assessment of Student Learning and 
Career Services. 

Actionable Results 

Combining the results of the Recommendations for Improvement with results against our 
external benchmarking groups, we created our action plan which focused on four factors: 

• Career Services (lowest performing, lowest comparison with external groups) 

• Quality of Instruction (L' Predictor) 



• Student Teaching Experience (2"‘^ Predictor) 

• Support Services Predictor) 

For each factor, we asked ourselves two questions “How important is this to our teaching 
goals?” and “How do we want to measure to see if we are meeting our teaching goals?” We 
concluded that the design of a functioning quality control system could not begin until a close 
examination of the current processes and policies used to develop curriculum and implement the 
education licensure programs had been undertaking. With these results, in part, the Program 
Chairs and a handful of administrative staff formed three committees: The Academic Advising 
Committee, The Academic Oversight Committee, and The Student Files and Records 
Committee. Each was tasked with unraveling the current academic processes and questioning 
the validity of outcomes. The overall responsibility and accountability for each area was 
examined and mapped. 

Next, the Program Chairs reviewed their curriculum. Course curriculum and student learning 
outcomes rely on the knowledge and skills students are expected to demonstrate. Each Chair 
mapped student learning outcomes to state and national standards. Program matrices were 
created for each program showing the relationship among program outcomes, Massachusetts 
DESE Professional Teaching Standards, national or Special Professional Associations (SPA) 
standards, TEAC Claims/Standards, course assignments, and key assessments. The information 
and data acquired from the assessment of outcomes was used in curriculum design and program 
improvements. 

Without these steps, no reliable quality control system could be identified or redesigned. We 
wanted to establish consistency, clarity, and confidence in our academic processes and identify 
the data to establish the internal audit and resulting quality control. Eike many institutions, 



Cambridge College has been implementing academic programs using a multitude of individual 
systems and processes reflecting historic precedence and individual preference, best practices 
created to address quality standards notwithstanding. 

Over this past year, faculty and administrators have worked tirelessly to create, improve, and 
update program and course outcomes, course syllabi, practicum, key assessments, admission 
processes, matriculation policy, academic advising, and handbooks. All areas were assessed to 
ensure consistency, clarity, and relevance to state and national standards while maintaining the 
mission of Cambridge College. 

Summary 

Assessment is critical to provide evidence of effectiveness and for program improvement. 
Creating effective assessments must be reliable, contain metrics that matter, and must drive 
action. Applying that assessment information to practice provides opportunities to improve 
programs and the student experience. 
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